[Bioclusters] NFS / SMC switches / GFS

[Steve Pittard]

On Sunday, August 28, 2005, at 11:42 PM, Joe Landman wrote:
>
> We haven't had much luck when pushing SMC and other lower end switches 
> very hard.  We have been using (and specing out) HP Procurves for the 
> last few years for smaller clusters (through about 64 nodes).
>
> It sounds like your network design is also a tree, not a fat tree, but 
> a basic tree.  This could explain the collisions (limited consumable 
> resource).  We usually spec this design for a maximum of 16 CPUs in 
> moderate IO situations.


Thanks for the reply. The SMC8648T , at least on paper, looks good 
(trunking, vlans,
88Gbs switch fabric, jumbo frames etc) though it does require rebooting 
with great frequency which
is not something I'm accustomed to seeing. The problem is related to 
job load which in turn
exacerbates underlying problems with NFS. We are pursuing more disks to 
improve performance
and implementing tuning measures for NFS but I'm not convinced that our 
setup is so far out of band
that a near daily reboot of the SMCs should be necessary.