Wednesday General Session

Today was demo-day in the general session. Nat did the SLED10 demo, highlighting glx & compiz, beagle integrated into gnome computer menu, and gnome file dialogs. They also showed f-spot and banshee, with ipod and digital camera plug-and-play.

They also demonstrated some workgroup functionality with OES Linux, including a cool pair of utiilities to migrate users from Exchange to GroupWise, and from a domain-based windows network into OES server, complete with group memberships, drive mappings, and data.

Martin Buckley also gave a demo of Identity Manager 3, but it was getting so late I had to go to my next session. I have loads of Identity Manager 3 sessions on my schedule for tomorrow so I’m sure I haven’t missed anything much.

Performance Tuning SLES

DL201 Performance Tuning SUSE Linux Enteprise Server

The session was about optimization of SUSE Linux Enterprise Server without losing vendor support and certification.

First make sure your hardware is suitable. That sets your maximum limit on performance. Ram quantity and bus are important. CPU speed is only critical for compute-intensive tasks. I/O, Bandwidth, disk, etc.

32-bit x86 cpus are limited to 20-40 GB of ram depending on platform. 64 bit units don’t have this limitation.

Use a separate disk for swap. Use mirroring and striping for performance on disk. Tune your hardware settings and disable what you are not using, especially in BIOS. You can usually tune for fast reboot rather than hardware startup checks if you need fast reboot.

Disable unneeded services. DUH, this is a primary one for security anyways, so you should be looking at this stuff anyways.

Consider traffic shaping for your application with wondershaper. This prioritises traffic on certain ports or protocols. That allows you to configure your server so that one user can’t hog all the network bandwidth, on a fileserver for example.

For checking io usage, install package sysstat and use iostat utility. The vmstat utility shows basic system usage. Top is also useful, and slabtop watches the slab cache.

Pick the right filesystem for the task. File sizes, number of files, workloads, and number of CPUs all affect the right choice of filesystem.

Reiserfs is very good for a lot of small files, like mail server, file server, database server, (not oracle), or other applications that use synchronous I/O.

Ext3 is good when you need a direct upgrade of ext2. It also works better than reiserfs on systems with more than four CPUs. When using ext3, enable btree support by going mkfs.ext3 -O dir_index. When multiple threads append files to a single directory, enable preallocation by going mount -o reservation on your ext3 filesystems.

XFS is good for huge servers (8+ cpus), very large filesystems (> 1 TB) and very large files.

OCFS2 is best for the Oracle Home filesystem no oracle systems.

You can use filesystem barriers on ext3 and reiserfs to improve data integrity but it costs in terms of performance. In SUSE it is turned on by default on both types of filesystems. In reiserfs use barrier=flush to turn it on and barrier=off to turn it off. The equivalent options in ext3 are barrier=1 or barrier=0.

Adjusting the journalling mode on reiserfs using the data= switch can affect the performance. Using data=journal can improve mail server performance.

You can move the journal of a journalling filesystem off onto a separate spindle with reiserfstune -journal-new-device /dev/xxx -s 8193. See man page.

Split filesystem workloads depending on access patterns. Put random I/O on separate spindles than streaming traffic. Consider disabling access time on files and directories for volumes that don’t need it.

Tune the I/O schedule using the kernel parameter elevator=. The first option is noop, best for smart storage controllers. Another is deadline, which limits the maximum latency per request to disk. The third is anticipatory (as), which maximizes throughput by increasing latency and is suitable to desktops, and completely fair queuing (cfq) which compromises between reads and writes and tries to balance throughput and latency, which is best for fileservers. The cfq setting is default in SLES.

Block Layer Tunables: There are lots of settings in the /proc filesystem. You use the sysctl command to tune this stuff. One to tune is the read ahead buffer in kb. It defaults to 128 but should be increased to 512 for raid or scsi storage.

Memory Management: Buffer flush daemon bdflush takes care of writing dirty pages to disk. You can tune it by several proc parameters in /proc/sys/vm/dirty_*. Look in /usr/src/linux/documentation to find out what all these parameters do.

You can change swappiness with /proc/sys/vm/swappiness. Default is 60. Lower values improve responsiveness when you switch between dekstop applications. Higher values let busy processes have more ram at the expense of latency of switching to rarely used processes.

Good documentation includes the source documentation, the admin manual for sles, for Oracle look at orarun to tune oracle, and /etc/sysconfig/ for special variables in your apps.

There is a system profiler called oprofile available for SUSE which can identify CPU hogs in the kernel or user-space. Look at the the sourceforge page for oprofile.

Novell Access Manager (iChain 3)

Session IO161: Novell Access Manager

Single solution to manage accss to web and enterprise applications. It supports federation to allow access to users from trusted external sources. It leverages web-based access to your enterprise applications to quickly allow acquisitions and mergers user-stores to access your stuff via federation. Adds sslvpn functionality to applications so you can authenticate via the portal in your browser and then use non-web applications with that single sign-on. Adds java agents for fine grained access control to web application server apps.

It has a flexible extensible policy engine.

Federation is based on SAML and liberty alliance stuff.

Access Manager comes with an identity agent that talks to an external LDAP store, including eDirectory, Active Directory, and Sun One directory server. It runs on a dedicated box.

It also uses an embedded eDirectory that it uses to store it’s configuration and policy. You don’t manage the embedded directory. It is all hidden and self managing. It does require a second machine, so to replace iChain with it, you need an additional box. One handles the identity stuff and the other handles the proxy stuff and ssl vpn (the Access Gateway). It sounds like you can have multiple proxy boxes but you do need that extra authentication server.

To use the ssl vpn, you login through the access gateway, and a little ssl vpn client applet or activex control downloads on the fly to connect you.

In Novell Access Manager you configure everything through a single web-based interface, instead of configuring each separate appliance one at a time plus the ACLs in eDirectory. You can group the gateways, so that you can apply a common configuration to a bunch of them at once.

OLAC is renamed to be called identity injection. This is the feature that sticks your single-sign-on info into the data stream when you login. You can have one policy apply to multiple protected resources so you don’t have to setup OLAC once for each protected resource.

They talked about liberty alliance bla bla bla federation bla bla bla. One thing it brings in addition to single sign-on and access control is single sign-off. You can also do account linking between one account to many, or one to one, and it allows you to limit what exact info comes across for each identity.

One of the big new features is support of the newest most elaborate Liberty Alliance spec that includes the ability to map info back and forth instead of just having single sign-on.

The Access Gateway supports NetWare or a SLES9 server as an appliance install. If you build it on Linux, you can have the sslvpn server, access gateway, and identity store piece on one box. You can’t do that on NetWare.

The ssl vpn is capable of client integrity checking before allowing an sslvpn session, “like is there a current virus scanner and a firewall,” and “is the application client version the current version?” It supports policy based authentication like groups, roles or users.

The java agents support sso on web applications servers JBoss and IBM WebSphere. They are also configurable via policy.

Provides SNMP, SYSLOG and SMTP alerting adn canned reports for Novell Access Manager usage.

This thing has a different license model than iChain. You have to buy a base pack which includes 500 user licenses and six instances of the access gateway. If you have upgrade protection for iChain for more than 500 users, you get the base pack plus however many more users you have beyond 500. If you have less than 500 users with upgrade protection for iChain, you are entitled to your user count of licenses plus only four instances of the access gateway (essentially four proxy servers).

Migrating a NetWare Cluster to an Open Enterprise Server Linux Cluster

The presenters were the authors of Novell Cluster Services for Linux and NetWare.

The point of the presentation was to move the cluster resources from an existing cluster running on NetWare to one on Linux without having to recreate the cluster resources.

In Linux clustering you use YaST to configure cluster nodes. There are similar but not identical cluster commands on Linux to those on NetWare. On Linux, you manage the cluster from iManager, but you can’t use ConsoleOne. You can also have Linux native filesystems as cluster resources. In SLES9 there is no iSCSI target, just an initiator, so if you want an iSCSI target you need NetWare. If you are using a SAN or shared SCSI you don’t need any NetWare servers for the cluster.

Supported filesystems for cluster resources include NSS, ext3, reiserfs, polyserve, or ocfs2. NSS has the advantages of having ACLs, inheritance, quotas, visibility, salvage, and large numbers of files, and huge filesystems. It is the best for shared storage for NCP clients, but reiserfs can also have all these features except quotas and salvage when shared via NCP on Linux. The disadvantage of NSS is that the metadata richness makes it a bit slower than reiser and some of the other ones.

When migrating to Linux from NetWare clusters, use NSS if you have existing resources. For stuff like iFolder, iPrint, AMP, GroupWise, and Instant Messaging, use Reiserfs because the features of NSS are not needed and reiserfs is faster.

Use reiserfs if a cluster aware filesystem is not needed. If a cluster aware filesystem is required, use PolyServe or OCFS2 (SUSE10). That will be available in OES2 based on SUSE10.

For GroupWise, do not use ext3 because it gets slow with large numbers of files which are generated by GroupWise.

For migration, you can do an in-place upgrade. You cannot do an in-place upgrade and maintain your configuration. You can do an over-the-wire migration and maintain your settings. You can also do a mixed-mode cluster by adding your Linux nodes to your cluster, migrate your cluster resources to the Linux nodes, and then decommissioning your NetWare cluster nodes.

The best upgrade scenario is to use SP2 of OES Linux and OES NetWare 6.5 and do a rolling upgrade by adding your Linux nodes to the NetWare cluster. Using a mixed cluster is not a permanent solution, but only really for migrating to a Linux cluster configuration.

After adding Linux nodes to a NetWare cluster, you can no longer add NetWare nodes. Also, some types of shared filesystems are not visible between the two systems until one or the other reboots, so it is not practical for long term. Also, the way Linux stores rights in nss is via the xml trustee file, but on NetWare rights are stored on the filesystem, so if you migrate back and forth the trustees get out of synch between the two. Also, cluster resources created on Linux won’t run on NetWare. Cluster resources created on NetWare are automatically translated for running on a Linux node but not the other way around.

To do a rolling upgrade, migrate all your resources off one NetWare cluster node. Then remove the NetWare node, install Linux on it, and add it to the cluster. Then migrate resources to another node and continue.

They did a demo where they built shared storage on an iSCSI target on NetWare and had two NetWare cluster resources pointing at the shared iSCSI storage. Then the migrated the resources to one node, installed Linux on the other node, and then added it to the cluster, migrated the resources, and then did the other NetWare node upgrade to Linux. The cluster resources were maintained in service the whole time.

Some guy asked about licensing. If you add a third node, you need a license. For Linux, you don’t actually need to have a license file. It works on the honour system. It still only includes a two node cluster license, but it is not enforced by the software.

If you have a problem with time synchronization in VMware virtualized Linux severs, you can use the grub menu.lst file to set a kernel parameter that says clock=pit, and then turn on clock synchronization in vmware tools, which should resolve the problem. This tidbit was contributed by an audience member. Another option if you can’t run vmware tools, is to set the kernel parameter clock=pmtmr, which makes the kernel more carefully correct for wonky timing, resulting in a more accurate clock. In SUSE 9x kernels, the default clock setting is clock=tsc, which aggressivly overcorrects for lost ticks, which makes the clock wildly inaccurate in SLES-based virtual machines. Here’s a vmware knowledgebase article on the subject. Google is my friend.

High Availability GroupWise

I went to a 2 hour session called TUT315 - How to Build a Highly Available Fault Tolerant Novell GroupWise 7 System

In a question at the beginning, the audience indicated that 1/3 to 1/2 of the room was running GroupWise in a cluster. That was a lot more than I expected and a lot more than a similar question that was asked in a session I attended last year showed.

The session talked about running GroupWise in a cluster. They mentioned that the problem with a GroupWise centralized cluster is that there is a single locale as a single point of failure. This talk also included GroupWise in a Business Continuity Cluster with nodes across a WAN connection in different physical locations.

Some key talking points were:

Domain and PO design
NCS clusters
Linux HA clusters
Novell Business Continuity clusters

I wanted to live-blog this class but the wireless network connection wasn’t working. I think it was because there was a Laura Chappel class going on at the same time and those guys were hogging all the bandwidth and running ethereal on 500 laptops on the wireless network.

The speaker mentioned that sometimes when there is a nework outage, the first thing you hear is that GroupWise isn’t working. Users say GroupWise is down, because they use it all the time.

Having a highly available GroupWise system requires highly available DHCP, DNS, SLP, and any other infrastructure pieces.

There is no need for a one MTA per POA relationship, unless you only have two POAs, in which case each one should be in a separate domain (unless the POAs are on one server, of course). Each gateway should also be in it’s own domain (or at least the gateway servers, if they run more than one gateway) should each have their own domain.

We should have two gateway servers. There needs to be more than one GWIA, each in its own domain. We could configure both with the same priority in the DNS mx records, so that they load-balance. One could be connected via a virus/spam scanner in Kelowna, and the other via one at the colocation site. That would split the traffic over the two gateways and provide the functionality of a backup mail server wihout needing a backup mailserver, and mail in and out would always be available. In GW7, GWIAs can be configured to be aware of each other and the system can automatically select one or the other if there is a service outage.

They (again) re-emphasized the 1 POA per server rule, except in a failure situation. My design for our new GW7 system is mostly validated.

POAs should not exceed 100 GB of storage. If more than that is needed, POAs should be split. This will impact our design, and I will have to repartition the storage in the bladecenter into 120 GB chunks, split between the two inital POA servers. This leaves the door open for another POA server to be added later as required. The reason for the rule-of-thumb size limitation is to allow indexing and maintenance routines to be accomplished within a reasonable amount of time. If we could convince the users to be responsible with their mailboxes, one POA could handle up to 700-800 heavy simultaneous users, but since they can’t bother to clean up, we hit the size limit before we hit the user-count limit.

The best and most powerful method to maintain reasonable post office sizes is to establish a mailbox retention and cleaning policy. This must be done by IT because generally most management-types don’t understand the issues enough to draft a policy. Then IT must take the policy to management, explain the concerns and need for a policy, and obtain buy-in from management, making it a corporate policy. The key issue is that email is a communications tool, not a data storage medium. The data that is sitting in the email system is not available to the appropriate users and is at risk of loss for corporate records.

GroupWise POAs do not need quad-cpu systems. POAs get I/O bound before they can utilize more than two processors.

On linux, Novell recommends using reiserfs as the data volume filesystem for GroupWise POAs. Other options are NSS or xfs/jfs. The ext3 filesystem gets performance bottlenecked when there are a large number of files and directories so it is not recommended for GroupWise.

Lots of info in this session, thank you very much. I have some re-thinking to do when I go home again.

Using SLE 10 as a MythTV Box

I went to a session about using SUSE 10 to setup a MythTV box, which if you don’t know, is a multimedia appliance for TV, videos, photos, music, and even VOIP telephony, built on all open-source software. The session was put on by Jeff Price from Novell, who is a very entertaining and engaging speaker. The content was pretty geeky and technical, which I enjoyed, and the demo was very compelling. The highlight of the demo (keeping in mind that this software can effectively replace Windows Media Centre Edition) was that as Jeff was demonstrating MythTV’s TV functionality, a Windoows Media Centre Edition commercial came on TV. Everybody laughed at the irony.

Believe it or not, I think there was real business value in this presentation. It would be a stretch to find some direct business use for a MythTV box, but you could probably do it. However, getting your technical staff interested in building a MythTV box at home, considering it can be a very technically challenging task, would have a lot of educational value. It would make your techie staff a lot more aware of what goes on under the hood in SUSE Linux.

SUSE Linux Enterprise - What’s New and Exciting

My very first session this year at Brainshare was IO 101, A Look Ahead, SUSE Linux Enterprise Version 10 - What’s New and What’s Exciting. Whew, the longest title in the Brainshare catalog I think.

We had just come out of the keynote where the announced some of this stuff, but in the session they went into it in a bit more detail. Some of the neat stuff:

The development process is like this: Open Source Communities around individual projects, where the projects are developed =&gt OpenSUSE project, where the projects are integrated, tested and bugfixed, with bugfixes going back to the community =&gt Novell SUSE Linux Enterprise, where everything is polished and made enterprise ready, plus supported for an extended period of time.

They announced that they would be giving DVDs of SLES and SLED 10 beta 8 during the conference. I got one afterwards, in the Novell Solutions Lab.

They covered a bit about Novell’s contributions to key open source technologies, including the Linux kernel, OpenOffice.org, Novell employs people who just work on open source stuff. They have also open-sourced some key formerly-proprietary technology like App Armor and YaST.

They announced support timelines for the various versions of SLES. SLES 8 will be supported until the fourth quarter of 2007. SLES9 will be supported until the end of 2011. SLES 10 will be supported for seven years, until about 2013. Subsequent versoins will come out approximatelly every two years and will be supported in a similar schedule going forward.

They said there are various improvements on enterprise hardware in SLES 10. It should be 10 to 15% faster on dual-core boxes. Multipath storage has been considerably enhanced. Xen is integrated in the default kernel, and managable directly in YaST. Hardware assisted virtualization is also supported (for Vanderpool technology in the new CPUs from Intel and AMD). Unmodified guest OSes such as Windows 2003 server or Solaris x86 will be able to run in Xen when hardware virtualization is supported in the CPU.

There is a new driver delivery mechanism allowing much easier packaging of vendor-specific hardware drivers. The drivers can be obtained from vendor websites and installed or put on CD/DVD and installed during the initial installation just like the vendor packages they had for NetWare.

Oracle’s cluster-aware filesystem, OCFS2 is integrated into the kernel, providing improved access to huge files with requirements for heavy random access. Within the next couple of years other applications besides Oracle will be able to be hosted effectively on OCFS2 partitions.

SLES 10 will support integrated clustering with up to 16 nodes using Linux HA system. The cluster nodes and resources are manageable in YaST. You will also be able to configure it as an iSCSI SAN, with the new iSCSI target software, which is added to the iSCSI initiator that already existed in SLES 9.

App Armor is integrated with configuration through YaST. About 15 preconfigured App Armor profiles will be included for popular applications, such as Apache, Tomcat, MySQL, Postgres, NFSd etc.

Management of patches will be done with a new unified Novell Customer Center, using ZenWorks technology. The server and client versions will be enabled for ZenWorks Linux Management out of the box.

Samba in SLES 10 will work as an AD PDC (not sure if I heard this correctly) so you would be able to have it participate directly in an Active Directory environment as a domain controller.

Finally, SLE 10 training from Novell is already available.

Good session.

Brainshare Blogging Day 1

As with last year, my Brainshare blog entries will be pretty much stream-of-conciousness stuff, as I am live-blogging or at least live-text-editing each session, depending on the network connectivity with the Brainshare Network. Today the wireless network was not quite up to snuff, and I couldn’t blog on the fly, so I wrote in emacs instead,

James saw this and said “Mecca!”

In the first keynote, Jack Messman and a couple of other executives from Novell gave presentations. A lot of corporate strategy marketing-speak was expounded. There was also a short demo of a few new and interesting features of SLES 10. There were a few nuggets of important information from our perspective.

The quick roundup of things that caught my attention in the first keynote are:

Novell announced support for NetWare until 2015 as a virtualized server running under Xen virtualization in a fully supported situation.

Novell announced the near-term availability of GroupWise Mobile Server, a system that provides GroupWise access to many many types of mobile devices including Windows and Palm OS smartphones. The software will be free of charge to existing GroupWise customers with upgrade protection.

Novell announced near-term availability of Blackberry Enterprise Server for GroupWise to provide access to GroupWise natively on Blackberry devices. The BES server will be free of charge to existing GroupWise customers with upgrade protection, and it will come with five free client device licenses.

Novell demonstrated Xen virtualization with VM configuration integrated into YaST on SUSE Linux Enterprise Server 10. They also demonstrated AppArmour, which ships in the box with SUSE Linux Enterprise Server and Desktop 10. There are in the area of 15 prebuilt application profiles in AppArmour for applications like Apache, MySQL, Postgres, OpenLDAP, and more. They also showed HA clustering in SUSE 10, with integrated YaST configuration of nodes and cluster resources in the GUI supporting up to 16-node clustering.

That’s about it.

Knee Still Works

James and I went for a long walk to Walgreens today, so I could get some Ibuprofen and a tensor for my knee. Then we went to the Salt Palace and got registered, and then strolled around the partner pavillion and talked to some of the vendors.

I found out that VMware Server will be supported for management but not in teh first release. I also found out that when a Vasco token shows “00000000″ instead of a six digit dynamic password, it means that the token has been deprogrammed by an electromagnetic field, that can happen when it is in close proximity to a device like a cellphone. They are not recoverable. We have a bunch of Vasco tokens, and we’ve had them in service for a couple of years. We’ve had two or three show up with all zeros in the display and we’ve always wondered what caused that.

Anyways, I iced my knee a couple of times today and it seems to be feeling better today. Hopefully it will continue to get better.

We went to the Welcome Reception in the evening. They had lots of food and entertainment, as usual. Here’s a silly picture of me that James took at the Oxygen Bar.

Kids Win Bronze in Waterpolo Tournament

The Waterpolo tournament ended today, and the kids won their last game, clinching the bronze medal. Way to go Mack, Emily, and the Thunderbirds. I think the kids will remember this tournament for a long time. It was the first time they did a whole-weekend away event with all their teammates in a hotel. They had a lot of fun and it was a great experience. Jenn says that all the families were talking about looking forward to the team and how it will be able to do even better next year.

I wasn’t too sure about waterpolo for the kids when we started, but it has turned out to have been a great experience for them and they met some good kids. It was well worth the time, cost and effort.