Copied on 30 Dec 2023 from here

There’s hundreds of threads on crossflashing variants of LSI 9211/9300/9305/9311/94xx cards here, on ServeTheHome, and on blogs. But they don’t always discuss the subtleties of doing it on an EFI board for users who may never have used EFI before, and there’s a lot of relevant background that “falls in the cracks” and causes frustration when it doesn’t go as expected. Some answers on the net are piecemeal, some are plain wrong; there’s not much solid, collated summary of the background knowledge as might be needed by a newcomer who wants to know enough, or to have a good chance that it “just works” without days of googling weird errors, and different recipes and software versions. This guide aims to explain “why” and “how” it’s as it is, and enough info on the tools and files for newcomers without experience. I cover EFI and non-LSI (OEM) variant flashing, so that even if the recipe doesn’t work, you know enough to do some debugging rather than google and try different flashing tool versions at random, and you know roughly how it needs to work and why. I’ve also updated it (2019-01-26,2019-02-16, 2020-01-29,2020-11-03) to cover card+controller information/resources and speed/compatibility/bandwidth “quick start” info, overheating/kernel panic/shutdown issues affecting some cards, and cable selection, for those looking to choose their cards, and a note on Supermicro HBAs. So it’s a bit more thorough than most howto’s - enjoy! I hope this helps someone out there. It’s my first guide here. I acknowledge the ton of information on numerous posts and blogs too many to mention. There’s also 2 open questions at the end, for anyone who wants to experiment and find the answers! Stilez INTRODUCTION The boards: LSI (now Avago/Broadcom) 9211/9300/9305/9311/94xx cards are widely recommended and used PCI-e SATA/SAS HBAs (host bus adapters). The most popular is probably the 9211-8i (“8 internal ports”) PCIe 2.0 cards. Very cheap OEM versions with virtually identical hardware are widely available new and second hand on EBay (Dell H200 PERC, Fujitsu D2607, IBM M1015, and others, full list+specs/notes at serverthehome.com). But the cards often needs crossflashing for several reasons: They need to be switched over to IT not IR firmware versions (“IT” firmware means it’s an HBA firmware which ZFS likes, while “IR” is a hardware RAID firmware which is strongly discouraged by most people for ZFS - the card can run either of these but often comes with the IR version loaded by default). People might also tend to prefer using LSI/Avago/Broadcom firmware not Dell/Fujitsu versions, which means changing the card’s recorded manufacturer. Last the cards often need updating to latest firmware. The process is easy given understanding and the right tools. But with so many slightly different recipes online and the flashing tools and process being pretty fussy about exact versions and steps, it often ends up with guesswork, random trial + error downloading, wasted time, and forum help requests. Not really ideal.

UPDATES:​

​

(2019-01-26): added new section on cards, chipsets and their variants at the start of the guide, with a link to a comprehensive resource list. It’s useful if you want to be sure which cards are out there, what their capabilities are, and how to make them work - especially PCIe 3 cards such as the more modern 9300/9305/9311/94xx.​

​

(2020-01-29) - added new sections on overheating/kernel panic/shutdown affecting some cards, and fanout cables.​

​

(2020-11-03) - added brief note on card/controller generational differences and mention of 94xx NVMe/U.2 cards.​

​

The user: I’ll assume you have (or plan to buy) a non-LSI branded 9211/9300/9305/9311/94xx (something like one of the cards named above), and quite probably an EFI motherboard, and you want to crossflash and also understand the ins and outs of the tools and process. If your motherboard won’t run EFI sas2flash it should run the MSDOS version - you’ll have to figure which one will run for you. I’m also going to assume you ‘ve read carefully online and haven’t found information you need, or you’ve tried following some guides and “it just hasn’t worked”. So this guide isn’t just a recipe - I will try to cover enough general info and background about the shells, the cards and the softwares/firmwares, so that if it still doesn’t work for you, you might be able to work it out and fix it anyway. Please add any comments what else you had to do, for other users’ benefit, to the comments on this thread. Downloads / Files: The files that most people will need, including the hard-to-find ones, are linked in a zip at the end (see ‘Resources’ at bottom of page). Unzip and copy to the root of your bootable USB stick and you’re good to go. If you want to be sure it’s safe, these were all searchable online and you can confirm that you found the right file (and that it’s safe) by checking hashes. Most of the inline links in the body, are to official download pages. I renamed some files in the zip file for clarity, because different versions all had the same name and it reduces mistakes. Obligatory disclaimer: As best I can tell this is accurate and will do what’s needed to crossflash these cards. However as with all other guides, YMMV. I’m not an expert. You have to judge it for yourself, I’m not responsible at all if your card doesn’t behave as expected or anything goes wrong (even if you follow this guide). That said, all my own 9211s are crossflashed exactly as described in this guide, and these cards are almost impossible to ‘brick’ whatever you do with the software provided (at worst, disable the boot rom from your motherboard setup, erase the card’s flash memory, reboot, and then choose between starting crossflashing again, asking for help with any errors, or reverting to the original manufacturer’s firmware) CONTENTS:

  1. Before you start: a) Known cards+variants, LSI chips/specs/capabilities, card/HDD compatibility, and PCIe bandwidth b) Supermicro HBA card notes (and possibly some other good quality makes!) c) HBA [“mpr”] overheating/cooling/kernel panic/shutdown issue affecting some cards d) Fanout cables (card to HDD/SSD)
  2. Safety precautions related to data and other hardware components
  3. Preparation - the basic software, and sorting out MSDOS/EFI and your platform
  4. Useful commands before starting
  5. Getting your card manufacturer’s IT firmware (if you don’t have it and it’s not in the attached zip)
  6. Firmware and flasher versions.
  7. Using megarec.exe and sas2flash / sas2flsh
  8. ERASING/WIPING THE CARD:
  9. OVERVIEW OF THE CROSSFLASHING PROCESS
  10. CROSSFLASHING DETAILS
  11. Other resources (including downloads)
  12. Open questions awaiting adventurous flashers!

THE GUIDE 1. Before you start: a) Known cards+variants, LSI chips/specs/capabilities, card/HDD compatibility, and PCIe bandwidth (added 2019-01-26) There is a useful and comprehensive list of HBA and RAID cards and their specs, from old to newest, here, sorted by the chip they use (2008, 3008 etc). Generally, cards in the same section are likely to be interchangeable, at least for simple card uses like HBA and well known chips like the 2008/2108/2208/3008/3108. For most NAS users, the card you want will be based on an LSI/Avago/Broadcom chip. LSI got taken over by Avago, and is now part of Broadcom. Everyone still talks about “LSI”, but cards or chips could have any of these names. They are the same company. RAID cards based on LSI chips run with “IR” firmware and in “IR mode”. Whatever the brand, they can almost always be crossflashed to operate as the equivalent HBA cards (“IT mode”), which means the card will ignore any cache/raid abilities, which is what you want for ZFS and HBA usage. So you have a wide choice. For HBAs, usually it’s just the chip determines the card’s capabilities. Two HBAs using the same chip almost always perform very similarly and use the same driver. Another giveaway is that their photos are often also extremely similar (because they reuse the manufacturer’s reference design, or rebrand their original cards).

Example: an IBM M1015 and LSI9211 both use an LSI SAS2008 chip, and they are in fact basically the same card, different brands. But there could be exceptions, so if unsure, it’s worth asking on the forum, if unsure whether your SAS2xxx/3xxx based card can actually be crossflashed and is supported by your system.​

This guide will focus on the 9211, which is cheap, very well supported, and incredibly widely used. It’s the most popular card and a good choice for a beginner. But the same principles do generally apply to the more recent LSI/Avago/Broadcom HBA cards and variants, such as the 9300/9311 (SAS3008/3108 chipsets) and newer 94xx. The crossflash method/details here mostly apply to those cards as well, so you won’t have to adapt much to get those on LSI HBA firmware and working with your NAS. CARD and CHIP (CONTROLLER) Generstions: For reference, the difference in cards and chip.generations is roughly - 2xxx series chips use PCIe 2.0 and up to 6 Gbps SAS. 3xxx series chips use PCIe 3.x and up to.12 Gbps SAS. The chios behind the 94xx boards (haven’t yet looked these up, just adding a mention!) can handle those, plus NVMe/U.2 devices for high speed SSD as well. HARD DRIVE/HBA compatibility: Note that some newer hard drives with updated protocols may NOT work on previous generation cards. Example: potentially some SAS-3/12 Gbit hard drives may not work with a SAS-2/6 Gbit card like the 9211. But as far as I know, that only affects SAS-3 and the upcoming SAS-4, and perhaps SATA Express/SATA-4 drives. It can be worth quickly looking up the HDD+card manufacturer’s documents, to confirm which SAS/SATA levels each of them supports. I haven’t come across a LSI 2xxx/3xxx HBA/RAID card that won’t work with SATA or previous generations, right now it’s mainly very new drives with older cards that seem to be mentioned. If that changes, please post a comment and I’ll update. PCIe compatibility/speeds/bandwidth: SAS2008/2108/2208 based cards are PCIe 2 8x cards, meaning PCI Express 2nd generation, with 8 “lanes” for data (each “lane” can provide a fixed amount of bandwidth, more lanes=more bandwidth). SAS 3008/3108 are PCIe 3 8x cards. PCIe 3 lanes can handle twice the bandwidth of PCIe 2 lanes. PCIe is very flexible - a PCIe3 card will work in a PCIe 2 slot at PCIe 2 speeds, or in a PCIe1x/2x/4x slot with fewer “lanes” and a reduced bandwidth limit. But they will work, and do the best they can, provided they physically fit. So you’ll often see something like “PCIe 2.0 4x in 8x slot” in motherboard specs, which means 4 lanes are provided, but the slot can accommodate cards designed for 8 lanes (at reduced 4 lane speed). If space is tight, it’s worth going for 8x or 16x slot sizes, even if the actual *lanes* are fewer, because at least your HBA card will fit! It’s hard-to-impossible to max out a 2008/2108/2208 based card with SATA 6 Gbit or even SAS-3 12Gbit HDDs. Bus traffic for SATA/SAS is likely to be nearer half than full duplex, giving a realistic bandwidth of PCIe 2 with 8 lanes used for HDD/SSD storage that is uncertain but at least 250 MB/sec ** per lane, so even attaching 8 HDDs all at maximum burst speed can’t usually do better than that. (** As best as I understand it, the maximum bandwidth quoted for PCIE assumes full duplex activity. But in practice when used for disk control, the duplex capability of SAS doesn’t lead to huge extra bandwidth use, for various reasons, and SATA 6Gbit doesn’t do full duplex anyway. Hence the system won’t be able to get full benefit of PCIE duplex capabilities and the realistic max speed could be significantly closer to half-duplex, or 50% of maximum PCIE bandwidth. Hence why the figure I’ve given as a safe minimum is 250MB/s not the expected/often quoted 500 MB/s. Please correct me if I’m wrong on this! Discussed in detail below) Generally if your HBA is handling rotating HDDs, even a 9211 is more than enough for 8 drives. The PCIe bandwidth sets a total, overall, limit, meaning that if say 4 disks are idle, their “share” of bandwidth can be used by other disks that are doing high speed transfers. SATA 3 SSDs are also limited to 6Gbit/sec. But if you’re using multiple SSDs or multiple SAS 3 HDDs, you might need to up your game, or put fewer disks on each HBA, or accept a reduced max total bandwidth. There’s a useful article on Wikipedia which lists bandwidths for various buses (PCIe 2/PCIe 3 etc) and storage protocols (SATA/SAS/NVMe). You can easily check if you’re pushing the limit. Add up the total bandwidth of all disks attached to the HBA, and compare it to the total PCIe bandwidth to the motherboard (lanes in use x bandwidth per lane) for PCIe 2 or 3, and for whatever number of lanes the card+slot have (or the lowest if they differ), to see if you’ll hit a serious risk of a bottleneck. b) Supermicro HBAs (and possibly some others) with “click and run” script (added 2019-02-16) Supermicro are a specialist server company. They use the same 3008 chip but their own board design, and from their reputation I’d trust their HBAs completely (check the recommendations for their motherboards on the main FreeNAS site). Also being own design, they are much less likely to be “grey market”, “knockoffs”, or “seconds” of LSI cards. Supermicro provide good clear drivers for IT, keep them up to date, and provide a full “click and leave” reflash script. If you get one of these, or are looking for 3008 cards, I’d add these to your search and I’d recommend the AOC-S3008L-8Le which is the 8 port HBA PCIe-3 version (equivalent 9300/9311), they’re often cheap among used 3008 cards. These are about the only cards I wouldn’t bother reflashing to LSI, although it would almost certainly be OK, because I trust their native firmware to be done well and up to date, hence no need. I am pretty sure their RAID cards (L8i) can be reflashed to IT mode using their own flasher as well, but you can easily check this with Supermicro tech support if you find one that’s cheap. Their firmware download contains IR and IT drivers and flashers in one ZIP, nice and easy to find. If you do get one of these, you can probably ignore the entire crossflash procedure below. Read the safety instructions, read the rest to get an idea of it, then follow their release notes and use their flasher and update the cards that way. There is a small issue with their flasher script. If you have 2 or more cards it will only detect + flash the first. To do the other cards, open the script in Notepad or a text editor, it’s got maybe 4 lines that start “!sas2/3flash”. Type these lines manually one at a time, but adding “-c 0” for the first card, “-c 1” for the second card, etc. When done, power off and reboot. End of flash. For everyone else, it’s a bit more work! c) HBA [“mpr”] overheating/cooling/kernel panic/shutdown issue (affects some systems and cards with more ports) (added 2020-01-29) The LSI HBA chips (2008/3008 and derivatives are used for a huge number of HBAs and RAID cards - both LSI/Avago/Broadcom and used in other cards. So your card, whatever its brand, almost certainly uses these chips. The problem is, LSI designed them with puny heatsinks, because it expected them primarily to be used in servers with forced air cooling. Result, these cards can drastically overheat. Reported temps of 100-110 C are common. The card wont be damaged - theyre designed to run hot and in the end the kernel will panic and the system shut down. But that’s not what you want. To be fair, most people don’t have this issue - they aren’t using a card intensely enough, or with enough ports. But those who do, there’s no real wayt round it. If you are seeing mpr device sensor showing high temps, and a possible eventual kernel panic, you’ll need to get a fan blowing on the card’s heatsink. It doesn’t need to be a big fan - a tiny one will often do. But you’ll need to get airflow one way or another before continuing. c) *Fanout cables (card to HDD/SSD) (added 2020-01-29)* A word of warning – get good quality cables! Do not skimp on those! Usually in computing, “any cable is pretty much as good as any other”, but I’ve had more issues with poor quality fanout cables for these cards, than anything else, and so have other people. Your best bet are cables by MicroSemi (used to be Adaptec or took them over). Personally I wouldn’t buy almost any other brand. Their part number is - For cards with SFF-8087 connectors (most 2xxx chips/PCIe 2.0 cards): 2275300-R (I think: no longer made but easily found online - google/amazon/ebay) - For cards with SFF-8643 (most 3xxx chips/PCIe 3.0 cards): 2280100-R These cables usually need MOLEX (old style 4 pin) connectors but most power supplies provide SATA connectors, so you might need to buy some SATA to MOLEX adapters/splitters, too. 2. Safety precautions related to data and other hardware components Crossflashing is pretty low-risk. This section addresses the three main areas of concern - hardware safety, data safety, and specific hardware/firmware based RAID and data caching/”disk speedup hardware” safety. Basic common-sense should apply when crossflashing and if in doubt it will not hurt to be a bit more careful. After all, you’re not going to do this every day. The short answer is that usually it shouldn’t be any worry, and you cannot “brick” these cards (render them unusable), but there are precautions you can take if concerned - and there are a few very definite warning signs to watch out for. The serious risks come with just two situations:

  1. Unplug anything with an “under the hood” hidden LSI controller that you don’t want flashed by mistake, such as some SSDs:) if you can’t unplug it, take great care not to accidentally erase or flash it. LSI controllers are often reused for other components, including as “hidden controllers” internal to some SSDs, and are also built in to some motherboards. Some SSDs include customised LSI HBA controllers internally to speed them up, which the flasher might detect and try to overwrite, and then you might have no working method to put back the correct custom firmware it used to have. Don’t assume what you have (unless extremely sure) - always check with the sas2flash -listall command to verify exactly what controllers the flasher software thinks it’s detecting, before erasing or programming any controller. *If the flasher detects more LSI controllers than you expect, when you do sas2flash -listall, then ** DO NOT IGNORE IT *** **
  2. If your system is actively using any kind of hardware or firmware based RAID, or disk speedup/SSD caching is in active use (especially if you plan to disconnect, flash, or change any of the cards or disks involved before/during the flashing process!), then check carefully what you need to do, to keep your data safe.

If these might describe your setup, read the next sections carefully. Otherwise treat the rest of this section as general information to skim over, and take what action you prefer. Hardware safety: Temporarily disconnect with care any other hardware or boot/data devices that might get scrambled by this, any drives you might accidentally destroy, or whatever. Especially, unplug anything with an “under the hood” hidden LSI controller that you don’t want flashed by mistake, such as some SSDs:). Note that some motherboards come with onboard LSI chips; if yours has one then take great care and ask around what to do, before going further. Data safety: In case you’re worried about your storage system - if you simply and temporarily disconnect the data cables from your entire FreeNAS boot + storage setup while flashing, then ZFS will happily find them all again when you reconnect them afterwards (provided the motherboard remembers its boot device or you remind it via motherboard setup when you’re finished!). The same is true for Windows and almost all other mainstream operating systems - if you tell it the boot device, it’ll figure out the storage, any ZFS volumes, dynamic disks, or other stuff, when it boots. The main exception where much more care is needed, is if you have any drives which are set up as a hardware/firmware RAID array or using any kind of hardware/firmware data caching or speedup system. (This is strongly discouraged with FreeNAS. If you’re using FreeNAS, hopefully you aren’t using HW RAID are you? :D Good - then skip to the next section!:rolleyes: ). This would includes all RAID arrays or SSD caching based on Intel or Marvel’s on-board chipset RAID and any other similar functionality. It also includes any hardware/firmware based disk caching + speedup systems, whether it’s SSD based or uses RAM or hardware card memory of any kind. This concern does not usually affect disk arrays that only exist within an operating system and has no hardware card or motherboard support, such as Microsoft’s Dynamic Disk based mirrors+RAID, or any ZFS volumes, because like I said, the operating system will almost always redetect the disks and resync the array when it next boots, and won’t make any assumptions about the state of data or how they should be connected. But if you do have hardware or firmware RAID/caching in use however, STOP RIGHT HERE and check what’s safest for your data before you disconnect drives or unplug controller cards. Quick checklist (just disconnect the drives, reconnect afterwards, let the OS figure it out):

  • Should be completely safe to “just disconnect” when powered down - Any ZFS volumes, any Microsoft dynamic disks, any individual disks which have no other RAID or caching in their data path and are just seen and controlled transparently “as they are”, by the OS. Hybrid disks with onboard NVRam caching shouldn’t be an issue since they are usually designed not to have their NVRAM cache seen by the OS, and any hard disk onboard cache will always be unaffected by disconnecting the drive from the controller.
  • Might not be safe to “just disconnect”; check first - Any hardware RAID cards or onboard systems that are in use and that expect disks to remain in a specific state across a reboot, or which expect cached data to remain the same across a reboot; any onboard or card-based hardware that might expect/configure RAID or caching during the motherboard boot process; any drives configured as hardware RAID using a hardware controller, any drives where Intel, Marvel, AMD or other firmware “soft” RAID is in use and providing caching or RAID (whether it was configured through an orom or from within an OS).

Bricking risk: It’s almost impossible to brick the 9211 variant cards or indeed any other hardware (i.e. make them unusable) with the files and actions we’re going to use here. A lot of online “howtos” say you can brick them by erasing them incorrectly or not saving the SAS ID first: they’re wrong. Put simply, you cannot render these cards unusable by erasing them the “wrong way” or not having a record of the NVRAM or SAS ID on them. See “Erasing” below, for more about this and how to recover from a bad flash or a 9211 card that’s interfering with the motherboard boot process. 3. Preparation - the basic software, and sorting out MSDOS/EFI and your platform Flashing software There are two main flashing utilities for these cards, “megarec” and “sas2flash”. Megarec.exe only works under MSDOS. sas2flash comes in 3 versions called sas2flsh.exe, sas2flash.exe, and sas2flash.efi, for MSDOS, Windows and EFI Shell. All 3 versions of sas2flash have identical commands, they only differ by what they run under. I would strongly suggest not flashing from Windows (less to go wrong!) and keeping to MSDOS or EFI. A small number of motherboards are pains in the neck - sas2flash detects EFI capability so MSDOS version won’t run, but it’s also not easy to get EFI shell to run either. If you have one of these boards, then come back to this guide once you’ve worked out how to get sas2flash to run - that’s too specific to each motherboard to cover here. Manufacturer and general computer support is probably a good place to start. You won’t have much choice which version of sas2flash you’ll use. If your motherboard appears to support EFI, then the MSDOS version won’t run (you can try but you’ll get a “PAL cannot be initialised” error) and you’ll have to use sas2flash EFI. If it doesn’t support EFI then of course only the MSDOS version will run. That means on an EFI board you’ll need to boot into MSDOS for some things and EFI for others. The EFI version of sas2flash was actually much nicer and easier than the MSDOS version in use, because EFI supports long filenames, command history, filename completion, and nicer command language You will need MSDOS for megarec.exe but EFI and MSDOS boot files can coexist on the same FAT32 USB stick and dual-boot on most EFI boards, and the zip file below is set up to dual boot both of MSDOS and EFI, so that’s no problem. You could run the MSDOS flashers off FreeDOS but generally blog posts suggest MSDOS seems more solid. Bootable tools for flashing I’m going to assume you will run this all off a USB stick, but you coiuld do the same off a CD/DVD or spare hard drive. The principle’s the same, you might need different software to create a bootable MSDOS device with MSDOS basic system files on a different type of storage device. Once your storage device is set up to boot and look for an operating system or shell, the LSI-related files in the attachment should be copied over to it, and also the MSDOS related files if it can’t find MSDOS. In case it’s unclear, the “LOCALE” folder is part of MSDOS, it’s not LSI related. 1. Get Rufus portable version ( https://rufus.akeo.ie ) and use it to create a bootable MSDOS USB drive (32GB max probably best for compatibility?). Any small stick is fine, the files are only about 8 MB total size. The only file systems MSDOS and EFI can both by default understand are FAT / FAT32 (*not* exFAT, NTFS or ext3/4!), but USB formatters almost always format the stick as FAT32 which is fine. If needed, force it to use one of those two. 2. Copy from the zip file, all the files under “USB” to the root of your USB stick. The download link is at the end of the page.

  • The files include a v1 EFI shell (compatible with all EFI boards; v2 EFI shells might not work on v1 boards) - note there are 2 copies of it because different boards expect it in different places (in the root dir or in \efi\boot). If you can’t boot to EFI check your motherboard bios/settings, and use any other EFI shell, including the motherboard’s own built in shell if there is one. It’s not important which EFI shell you use, so do whatever’s needed to get into the EFI shell and have those files accessible on USB. ( Shell download site )
  • There’s also a file “DOS4GW.EXE”, which megarec.exe needs to run and complains if missing. It’s easily found online and I think comes from unzipping FreeDOS. Like the EFI shell, there are also two copies, one in the USB root and one with the Megarec.exe file, because the error refers to \dos4gw which suggests it’s looking in the root directory. It was easier to add 2 copies than check what Megarec needed :)
  • Last there are 2 folders, one has Megarec and the files needed with it, one has sas2flash and the files needed with it. There are different versions of sas2flash in there, you need them. They’re renamed to show the version of the flasher and its manufacturer. You shouldn’t need other flashers - LSI’s flashers will flash anyone else’s roms and firmware nicely - but you might need to google and grab other versions of the actual firmware such as Fujitsu’s version of the IT firmware for your card. I’ve included the Dell IT firmware which worked on my H200 PERCs.

3. Once you’ve done that, you should be able to boot in MSDOS and also if your board allows, into EFI. You usually do this by selecting the boot menu. On an EFI board the USB stick will show up as 2 options: “USB (NAME)” and “UEFI USB (NAME)” - choose the first for MSDOS and the second for EFI. They look completely different (EFI has colour!). On my ASRock test system I hit an oddity that it wouldn’t load EFI shell from the boot menu but would run it if I selected “Run EFI from filesystem” from the setup rom’s exit menu. On my SuperMicro board I had to go into “Boot options” -> “EFI Boot Order” in the firmware, and manually add “Boot to EFI Shell” as an option, before it would let me boot to EFI shell. On some systems EFI shell startup may be case sensitive and complain at “shellx64” not being called “Shellx64” or whatever, and on some blogs the shell is said to be named “bootx64.efi” not “shellx64.efi”. If you rename it, remember to rename both copies (in the root folder and in \efi\boot). This guide will work whether you have one or multiple 9211-style cards installed, but it’s not unreasonable to do them one at a time. I got 3 cards as a job lot online for £15 and did them all at the same time, and it was fine. 4. Getting your card manufacturer’s IT firmware (if your card is running IR firmware, you don’t have the manufacturer’s IT firmware, and it’s not an LSI 9211-8i/e or Dell H200 PERC 8 port card, which are in the attached zip) 9211 firmware comes in an “IT” and “IR” versions. I cover the difference between them below, but for the moment what matters is that if you’re using the card with ZFS, as FreeNAS does, you will almost certainly need to have the manufacturer’s IT version firmware (for HBAs) not the IR version (for RAID) during the crossflash process. If your card comes with IR firmware not IT (or if you aren’t sure which it’s running) then you’ll have to download and unpack your card manufacturer’s version of the IT firmware, if you don’t have it already and it’s not already in the attached zip file. How do you tell whether or not your card is already running IT or IR already? If the manufacturer/vendor doesn’t make it clear then there isn’t a clear-cut way. One way is to boot the computer normally and enter the card’s boot rom (orom, option rom, etc). If it shows loads of options and advanced RAID configs such as RAID 5 or caching, it’s almost certainly IR version. If not, it might be an IT version but it’s not certain. You could just try the flashing instructions and see if you get a “cannot flash IT firmware over IR firmware” error. Otherwise, the easiest answer is to assume your card has IR firmware and go track down the card manufacturer’s IT firmware, unless you know differently. I’ve included LSI and Dell’s 9211 IT firmware in the zip file, and most IBM M1015 are already supplied with IT firmware installed, so with luck you won’t need to look hard, or your card manufacturer will make it easy to find and download (ask online if not). With luck you either have a card whose IT firmware is in the attached zip, or a card that’s already running IT firmware. However in a few cases, you might need to find and download the IT firmware for your card’s manufacturer. In a very few cases, the manufacturer only supplies their firmware bundled as part of an EXE, ISO or other file type. It might also be packed as a file for a different card - for example Dell only supply 9211 IT firmware with their 9211-8e version card, not their 9211-8i version card. But it’s the same hardware and works anyway. If that’s your situation, you will need to find and download the firmware, and also perhaps find a tool to unpack it, so you can get the actual IT firmware file. Tools for this include WinRAR or 7zip. If you’re in that situation, then this should be pretty quick and easy to do. Google is sure to have information to help you as well, and if not, ask on this or other relevant forums. 5. Useful commands before starting Useful EFI shell commands (for those not used to EFI shell): help, ls, cd, cp, mv - similar to basic linux (ls = list dir contents, cd = change current dir, mv/cp = move/copy files) map - lists disks. use “map” to quickly see which fs…: device your USB stick is fs0: fs1: etc - select a device (Typing fs0: selects the device shown next to fs0: in map as the current device/folder. Like typing a drive letter at the MSDOS prompt) Paths - directory paths use MSDOS style “\” as a directory separator not linux “/”. Tab - usually does filename completion. -h, -help, –h, –help, ? - most commands will have basic help (if not, Google will help instead). Try these after the command, to get quick help on commands. -b - most commands accept this option to show the output page at a time (try “help -b” or “map -b” for example). It’s useful if the command shows so much that it scrolls off the top of the screen. If “-b” doesn’t work you could redirect the command’s output to a file, then open the file in EFI’s text editor (or even save to a second USB stick and open on another machine if everything else fails). You’ll have to look up the details how to do this, though. You don’t need more than that for this guide. Useful sas2flash commands: sas2flash -listall - lists all LSI 2008 controllers found. This is the controller in the 9211. Ideally equivalent to listing all 9211 variant cards found, but if it finds additional controllers DO NOT IGNORE THIS! See above - occasionally extra LSI 2008 controllers may be hidden in SSDs and on the motherboard. sas2flash -c CARD_ID -list - provides full details about each card (current firmware, bios, versions, sas ID, manufacturer, whether IR/IT, etc.) 6. Firmware and flasher versions. Now we dive into the firmwares and flashers. This is where the fun starts, but it’s essential if you want to understand why the instructions are so darn messy and why they often don’t work for people first time. CARD_ID FOR FLASHING PURPOSES: Both Megarec and sas2flash refer to cards by a CARD_ID. This is basically how they were enumerated at boot, so 0 = first 2008 based card it found, 1 = second 2008 based card it found, and so on. They are always numbered 0 upwards, whatever’s in your box at the time. The CARD_ID is actually all you need to access the card and program a firmware and SAS ID, and it’s how the flashers (all of them) identify which card to flash or program a SAS ID to. See “erasing” below for more on this. SAS ID: The 9211 variant cards have a SAS ID in their NVRAM (technically called the “WWM”). It’s a straight 16 character hex string, sas2flash -c CARD_ID -list will show the card’s SAS ID if assigned. Two cards in a computer must have different SAS IDs. (Technically it should be worldwide unique, like a MAC address, but most users here probably don’t care about that). Many web pages state that if you delete or lose the SAS ID, or erase then reboot without making a note of it, then you’re screwed, but that’s incorrect, because you can always access the card by its card ID and you can always write a new SAS ID later (or invent a SAS ID at random if you need to!). The only time it will really matter is when you have an existing system that relies on that ID. But if you’re crossflashing - and especially if you’re going to use an HBA/IT firmware, and double-especially if you’re building a FreeNAS box that doesn’t check whose brands are in the box - you probably don’t have that situation. So take a note or photo by all means (I’ll explain how to find the relevant info) but if you don’t, or you can’t be bothered, it’s probably (=almost always) fine. The card may have the SAS ID on it somewhere, if you look for it - on a small label or whatever, if it matters. If not, use the above command first of all, to get the info, if you insist on making a note beforehand :rolleyes: But if it’s a “clean start” situation with no need to preserve existing SAS topology or attached disk info, mostly you just want to be sure the SAS ID is unique in that box and you won’t need to have an existing SAS ID to crossflash+prepare your card. Again worth noting that it doesn’t usually matter what value you give it, and you can update its SAS ID at any time after flashing by booting into MSDOS/EFI and using sas2flash -c CARD_ID -sasadd SAS ID. The only requirement is no other installed card has the same SAS ID. Note that sas2flash -list will display the SAS ID with hyphens, but sas2flash -sasadd expects it entered without the hyphens as 16 plain hex characters. If you do erase the SAS ID (WWN), then you’ll need to reenter it or another SAS ID at the end of flashing, using sas2flash -c CARD_ID -o -sasadd SAS_ID. To mention again, usually the original value is on a sticker somewhere on the card or motherboard, but if not, make a note or even easier, just pick (almost) any 16 hex digits at random. FLASHABLE SOFTWARES (“BOOT ROM/BIOS” AND “FIRMWARE”): Confusingly, the 9211 cards have their flashable files in *two* parts. There are bios and efi “roms” that provides boot-time options, and these are separate* files independent from the “firmware” (the part that actually controls the disks and responds to the OS). For the boot roms one can flash a BIOS orom, an EFI boot rom, or neither or both (as they can coexist). For HBA use such as FreeNAS, only the “firmware” part normally matters. You don’t need to flash the orom or EFI rom. The boot roms are completely optional and while nice, they cause the boot to be slower and they don’t add anything really. It can be flashed onto the card later if you ever change your mind. To repeat: you do **not** need to flash the bios/efi boot roms to get a fully functional HBA card. You only need to flash the firmware to get the card working fully. The only time you usually *need to flash the “rom” firmware (orom/EFI/bios) is if you want to boot the system itself from the card, and the disks attached to the card must be visible at early boot time to do so. Also for some card settings such as built-in staggered spinup or card enable/disable. But because most users use these cards only for attaching data disks, very few users/systems will ever need to flash the “rom” part of the software. For everyone else, they just slow down the boot process, and the “firmware” part of the firmware is all you’ll need. (If you’re interested, the bios/boot roms mainly seem to provide features such as exposing disks to the BIOS/firmware during the boot process (if booting off a disk attached to the card), or showing card ID, SAS topology, list attached devices at boot, and enable/disable. They also allow you to set some options for removable media, card boot order if you have >1 card, number of interrupts used, etc. Most of these options aren’t really needed for FreeNAS or for an HBA, especially since FreeNAS itself will tell you what’s attached when it boots.) **FIRMWARE/BOOT ROM VERSIONS, VARIANTS AND THEIR USUAL FILENAMES: The firmware file itself comes in two variants, depending if it’s to be used as an HBA (the “IT” version) or a RAID card (the “IR” version). For FreeNAS you almost always want the IT version of the firmware, but the 9211 variants mainly seem to come in IR versions so they will need cross flashing, to replace the IR version by the IT version, as well as being flashed again to switch from Dell/whoever’s firmware to LSI’s firmware builds. The firmware most people will want to get working is LSI/Avago/Broadcom IT software (currently version P20). It may also be known as the firmware for HBA not RAID cards, but a dell/fujitsu card will usually have dell/fujitsu firmwares as well and you’ll probably need that as well. The support website or Google will help you find your own card’s firmware. (See “crossflashing process”). The firmware and boot rom files, and sometimes the flashers, are often wrapped up in zip or exe files which need to be unpacked and the relevant files extracted. WinRAR, 7zip or other unpackers that can unpack .exe files can be useful for this. You will usually need your card manufacturer’s IT firmware for this process, even though you only need it temporarily and you’ll overwrite it with the LSI firmware almost immediately after flashing it. (I’ll explain why below: basically there’s a step where LSI have a “lockout” that this helps you to work around). IMPORTANT NOTE: when you go looking for your Dell/IBM/Toshiba/??? IT firmware, you do not need the latest version. Any version, however old (almost!) will do. In fact older versions may be easier to flash. Don’t worry at all if it’s outdated, since you’ll overwrite it a few minutes later with the current LSI firmware. It’s only needed for a temporary step in the process. Filenames for the firmware are usually “2118it.bin” (for LSI IT firmware) and “6GBPSAS.FW” (for Dell firmware), whatever version they are. I haven’t checked what filenames IBM, Fujitsu and other OEMs usually give their IT firmware versions. LSI make their IT versions easy to find. Dell do it like this: the firmware for their 9211-8i cards is usually IR version (“i”=internal ports, presumed to be RAIDed), and the firmware for their 9211-8e cards is usually identical but IT version (“e”=external ports, presumed to be HBA). So if you have a Dell card like the H200 or H310, instead of looking for an IT version for the 9211-8i look for the firmware for the external port version (equivalent to the 9211-8e). Dell have an A10 version of the firmware titled “SASHBA_Firmware_6GBPS-SAS-HBA_07.03.06.00_A10_ZPE.exe”. Expand the .exe file and grab the “6GBPSAS.FW” file inside it, that’s the Dell IT version you need for the 9211-8i or 8e. Once again, if it’s some other manufacturer, search their support section, ask on their forums or here, or Google it. But in any case you want their IT (HBA) not IR (RAID) firmware version. If you want to flash either or both boot roms, their standard filenames are x64sas2.rom (for the EFI boot rom) and mptsas2.rom (for the BIOS boot rom also known as option rom or orom). LSI calls their EFI boot rom a UEFI BSD rom (not to be confused with FreeBSD). You can flash either, neither or any combination. But as I’ve said, you don’t have to, and it won’t do much for most people. 7. Using megarec.exe and sas2flash/sas2flsh As I mentioned, there are 2 utilities - megarec and sas2flash (AKA sas2flsh). Megarec uses MSDOS while sas2flash has both MSDOS and EFI versions. This is a pain, because sas2flash for MSDOS (.exe) will NOT work on a board that’s detected as EFI, and sas2flash EFI will NOT work on a board lacking EFI. So you have no choice about whether to use MSDOS EFI versions, it’s determined by the motherboard you’re using. If you get an error about “unable to initialise PAL” when you try to use sas2flsh.exe in MSDOS, that means it’s trying to use the MSDOS version on an EFI-detected motherboard. Switch to the EFI version instead. This mixed MSDOS + EFI environment isn’t an issue (i.e., MSDOS for megarec and EFI for sas2flash) because a single FAT32 USB can contain both MSDOS and EFI boot files, and boot either on demand, so no need to have 2 sticks for this. See above for how to build a bootable USB stick. The zip file attached to this resource is already set up to allow booting both EFI and MSDOS if your board allows it - just copy its contents to the USB stick root directory and choose which one to boot to. Using Megarec.exe: Megarec can run off any MSDOS disk on any motherboard. It needs to find the system file DOS4GW.EXE to run, but that’s easy to find online (I think it may be part of FreeDOS). Megarec has just two uses, but they’re important. It reliably erases the card (less fussy than sas2flash), and it also backs up/restores the card’s SBR sector if you need that (you probably don’t). It’ll do this even if sas2flash has screwed up the card flashing/checksums and gives errors when you try and erase it. So it’s used to put the card in a known state before crossflashing or if it’s screwed up and unflashable, or to backup/restore the SBR data. Everything else needs sas2flash / sas2flsh. Go into MSDOS, use it, quit MSDOS. See below for more on how to erase the cards properly. Useful Megarec commands (see below): megarec -readbr CARD_ID FILE - read and write sbr data megarec -writesbr CARD_ID FILE megarec -cleanflash CARD_ID - erase the card As always the card ID is based on whatever cards are in the box at the time. They start at zero, so if you have one card installed it’s always called card 0, the second card is always called card 1, and so on. So to erase the 1st or only installed card, “megarec -cleanflash 0”, and so on. sas2flash uses the same IDs. Megarec allows the backup and restore of SBR data on the card. Most people won’t need this, and you’ll have to Google it if interested. One of the links at the bottom of this page gives information on SBR and notes that if the card sees some but not all drives when crossflashed, the SBR data might in some cases be responsible. I have never seen this issue myself and have never needed it, but YMMV. If cautious, perhaps back it up before anything else “just in case”, so you have the original to work on if it turns out you need it. But most people won’t. Using sas2flash (a.k.a. sas2flash.exe, sas2flsh.exe, sas2flash.efi - they all have identical commands) Where I write “sas2flash”, you may have to type the command in full (sas2flash.efi not just sas2flash) sas2flash has a quite rich command set, but you don’t need most of them for this. The main command line options are: -c CARD_ID optional if sure there’s only one LSI controller present -o enables “advanced mode” for that command. Although “-o” does nothing by itself, it’s needed as a safety measure when using certain sas2flash commands. If this option is omitted when it’s required, some commands will not work, or will fail midway. Check “-o” isn’t omitted if a sas2flash command you try doesn’t go as expected! Advanced mode commands used here: sas2flash -c CARD_ID -o -f FILE = flash a firmware file sas2flash -c CARD_ID -o -b FILE = flash a boot rom (bios or EFI) file sas2flash -c CARD_ID -o -e SECTION_ID = erase some part of the card’s NVRAM (see below - important!) sas2flash -c CARD_ID -o -sasadd SAS_ID = update SAS ID (see above) sas2flash -listall = lists all cards found. Again, technically it lists all LSI 2008 controllers found. If the number of items listed isn’t exactly what you expect, STOP IMMEDIATELY AND REREAD ABOVE. Make sure you don’t accidentally flash an SSD or motherboard (or other hidden) LSI 2008 controller before continuing. sas2flash -c CARD_ID -list = get full details about each card, including current firmware, bios, versions, sas ID, manufacturer, whether IR/IT, etc. SAS2FLASH often reports errors that can be ignored. For example it tries before and after each command to disconnect/reconnect the EFI driver - but if no EFI driver has been installed it gives an error for both of these (“unable to connect/disconnect EFI DRIVER”) which can be completely ignored. Also (and especially on non-LSI firmware) it reports things like “flash succeeded” and then error messages such as “due to errors, remaining commands were not executed”, which can be ignored if the main task was okay but are ambiguous at best. sas2flash can also take a long time to process some commands. I have no idea how it can take 40 seconds just to figure that “-c 0 list” isn’t a valid command (missing ‘-‘ before list) , but it does. So if it takes 30 - 60 seconds to get going on a command, that’s just how it is. Write a few tweets and come back in 30 seconds. Reboots aren’t normally needed between flashes, but it might not hurt in some cases. Where it seems to matter, I’ve noted it below. I wan to emphasise here, that sometimes sas2flash seems to get FUBAR’ed. If at any time during flashing, you find that basic operations like sas2flash -list or flashing aren’t just misreporting but are unable to complete any more, and it looks like it’s because the card is scrambled or bad NVRAM checksum issues, not just because it’s a wrong version of something somewhere, or what it’s found isn’t what it needs, and you can’t figure it out, don’t hesitate to go back to wiping the card with megarec and starting the crossflash again. I’ve assumed below that you are using EFI. If you aren’t then the same should work for MSDOS using the P5 MSDOS flasher instead (also included in the zip), and do everything else the same. 8. ERASING/WIPING THE CARD: Most guides say to use sas2flash -c CARD_ID -o -e 6 (or “7”) to erase the card. But having played with sas2flash a fair bit, I wouldn’t do so. I did *not* use sas2flash for erasing. It just wasn’t reliably effective for me, in the sense that some versions of sas2flash worked but some didn’t, some worked on one firmware not on another, and I wasn’t always convinced it had or hadn’t, or whether to use a different version, etc. The crossflash process was complicated enough without that confusion. For example sas2flash EFI P5 said it hit an error on erase (-o -e 7) at some points where megarec was fine. Sometimes it seemed to part flash and screw up the checksums, so I had to re-erase or other steps failed, and when that happened I wasn’t sure whether it was the erase or the next step, which was responsible for the issue. Basically some versions of sas2flash wouldn’t erase everything, or seems to not work, or it was dubious. Once I discovered that not only megarec was more reliable and explicitly verified (on all manufacturer’s versions), and also that I didn’t need to preserve the SAS ID like some guides said, so I didn’t need to distinguish erasing with -o -e 6 or -o -e 7, it was easy. I used megarec to erase the cards and it worked first time, every time. I recommend you to use that instead, as well, and ignore the guides which state to use sas2flash -o -e 6/7 unloess you have some particular reason to do so. In fact I just don’t use the sas2flash eraser at all, if I need to erase I reboot MSDOS and use megarec. I found megarec -cleanflash to be much more reliable; it clearly worked each time and there’s a visible verbose percentage counter to watch, and no spurious error messages. The only downside was loss of NVRAM data such as SBR and SAS ID, both of which can be backed up or aren’t important, and really are expected in a full erase. Erasure + bricking worries info: Some websites warn that erasing the card and rebooting will leave it inaccessible / bricked (OHNOES!). Some say it’s a risk to reboot without reflashing some firmware at least, or without a note of the old SAS ID. So far as I can tell, both of these are just internet myths. That’s not my experience at all. This is what I’ve found: I’ve been using megarec to fully erase the cards, and rebooted, then used sas2flash to reflash the erased card with the new firmware after it’s rebooted. It worked just fine. It seems you don’t need the SAS ID or anything else in firmware to be able to reflash new firmware or add a replacement SAS ID to the erased card after reboot. Just boot the system, use sas2flash -c (0, 1, 2 etc) -o -f FILENAME to flash the new firmware on the erased card, whether in EFI or MSDOS, add a made-up or previously recorded SAS ID using sas2flash -sasadd HEX_STRING as described below, and carry right on. If the card or motherboard seems to become unresponsive or bricked because of the flashing process, these are the recovery instructions, but I should say I have never to do this, it’s never been that bad. These cards simply cannot be messed up so badly that megarec can’t low-level erase them, as far as I can tell.

  • If the motherboard itself won’t boot, then * Turn off power and remove the card. * Reboot. * Enter your motherboard BIOS/setup panel and disable orom on boot for storage cards or for that specific card, if enabled (so the card’s misflashed state can’t interfere with proper booting). ORom enable/disable is often buried under the “boot” menu, or in some kind of “CSM/compatibility” submenu, but may be elsewhere. * Save motherboard settings, exit setup, turn off power again and plug the card back in. * The motherboard will now boot, as the orom won’t be relied on or executed at boot-time.
  • Then in any case, * Reboot MSDOS and erase card’s NVRAM using megarec as described above. * Reboot. * If you disabled oroms above, then you can optionally re-enable them again now. Solved. Card will now be erased and flashable, and the system will boot as normal. Carry on with your crossflashing process as described below.

9. OVERVIEW OF THE CROSSFLASHING PROCESS The crossflashing process has some twists to it, and this is usually where it goes wrong for people. Before giving the detailed steps and commands to use, here is an overview of what we will be doing. It will give an understanding of the actual crossflash process, and why the “recipe” below is as it is. - On these cards, you can’t usually flash an IT firmware over an existing IR firmware. You have to erase the flash, then flash from IR to IT of the same brand. Only once you’ve got it running the *IT* firmware can you make the switch to LSI if it’s a Dell or other manufacturer card. (When I say “Dell” I really mean “whatever your card brand is”). - These cards also check who the manufacturer is, and from about version P7 of the LSI flasher, they blocked the flash process if you tried to replace the firmware by equivalent firmware of a different manufacturer (e.g., overwrite Dell with LSI). It will remember its “manufacturer” even when fully erased, until a new manufacturer’s firmware is flashed. So even once you’ve flashed from Dell IR to Dell IT firmware (or whatever brand it is), you still can’t jump directly to LSI latest IT firmware using a recent flasher, because the recent flashers just won’t let you. So the “big picture” process for going from Dell IR to LSI P20 IT, is something like this:

  1. Grab the SAS ID (and perhaps the SBR data) if you care about it, check your system is okay and any storage devices safely dealt with (see “preparations”), and download/unzip a known good version of your card’s OEM (Dell/Fujitsu) IT firmware to use as a first “jump”. The IT firmware should be easy to find on your card manufacturer’s download site; if not then Google it. It’s the same firmware line as already installed, just the IT version. LSI’s P5 flasher will program the manufacturer’s early version IT over IR regardless of the make of card you have. It doesn’t matter too much what version you find. As long as its the same manufacturer the LSI flasher will generally let you flash it after you erase the card first. But on the whole, earlier versions are better than latest versions for this bit, if you can find them easily, since the most consistently working LSI software for crossflashing is the early versions before about P7.

  2. Check the flasher shows the number of LSI 9211 controllers you expect it to. LSI controllers can be hidden internally within other hardware such as some enterprise or high-spec SSDs or built into the motherboard. We check before going further, how many LSI controllers the flasher detects, and that it’s the same as the number of LSI controllers you expect it to find, so that nothing gets wrongly flashed if there is a hidden controller anywhere.

  3. Erase the card’s NVRAM. *This is a place things often go wrong for people following poorly written guides or erasing non-LSI cards with sas2flash - not meaning that you brick the card, but that the card refuses to erase as expected, or isn’t properly erased, or you can’t be sure it’s really been erased, or you get an odd error during erase. Some guides also leave people in doubt what’s “safe” or “best” to erase and how much should be erased (Should I use -o -e 6 or -o -e 7? Will I brick my card? HELP!). The common online suggestions are not (in my experience) usually the best or safest way. Read the detailed sections above (erasing) and below (crossflashing) carefully and make your own choice. You need certainty that the erase process has worked properly, otherwise you’ll probably get weird errors or failed flashes in the next steps.*

  4. Even when erased, the card will still “remember” it’s an IR card, so we deal with that next, by flashing your manufacturer’s IT version firmware onto it, if not already IT. You need to use a flasher that will do this properly. The LSI P5 EFI flasher (and MSDOS version if using MSDOS) seems to be very reliable for other OEMs and versions, but if not, you’ll have to find a version of sas2flash that does work for it and for you. At a pinch the card manufacturer probably has a flasher to switch firmware to IT mode, if you just can’t get LSI’s to work. Contact their support forum or tech desk. You do *not* need to worry about flashing the bios/efi rom at this stage, it can wait until you’re on LSI P20 if you want to do it at all.

  5. You now have a Dell (or whatever) IT card. Yay. Now we need to flash a second time, this time to crossflash it to LSI. Use the same LSI P5 EFI flasher (again!) to flash it a second time, this time taking it from Dell IT to LSI IT version P7, which is roughly where LSI started to lock down the manufacturer ID. The early flashers such as P5 (but not more modern ones!) will ask if you’re sure you want to change manufacturer ID: obviously “Yes”. Don’t forget the “-o” option or you won’t see this prompt.

  6. You’re now on mainstream LSI P7 IT firmware, so manufacturer lockout is dealt with. That means you can go directly to using the latest flasher (P20 right now), flash a *third* time to install the latest LSI IT firmware, and give the card a new SAS ID (either same as before or made up). At that point you’re done.

  7. Flashing a BIOS orom, or EFI boot rom, is an optional extra if you want. You can do it, too, but it won’t do much on a FreeNAS box other than slow down your boot.

In other words flash 1) DELL IR -> DELL IT, 2) DELL IT -> LSI IT P7, 3) LSI P7 -> LSI LATEST (P20+). The main reasons flashing doesn’t work for people are:

  • odd error messages that take time to work around
  • version selection of flasher at step 3, and flasher + firmware at step 4.
  • LSI’s flashers and cards are very fussy about which version will or won’t work, and are happy to throw up confusing errors or not behave as expected, if they don’t like your choice.
  • not erasing the NVRAM properly when it’s needed (almost always do a proper full erase before you do any crossflashing at all, even if it has valid firmware when you first begin)
  • forgetting the “-o” option in some sas2flash command

10. CROSSFLASHING DETAILS Finally here is a recipe. But with luck the previous information will help a lot to make sense of it, and to figure out what to try if it fails, and why it might fail (if it does).

  1. Boot into MSDOS/EFI as needed for sas2flash on your motherboard. You can get the SAS ID and other card firmware info if you want it, at this step (not essential, but doesn’t hurt). However the important thing is checking there aren’t any hidden LSI controllers, before we do any erasing or flashing. sas2flash -listall (TO LIST CARDS) sas2flash -c CARDID -list (TO GET SPECIFIC CARD DATA) This 2nd step is optional but not a bad idea. You can also confirm the firmware, brand etc from the same output. SAFETY WARNING: IF THE OUTPUT FROM sas2flash -listall SHOWS MORE LSI CONTROLLERS THAN YOU EXPECT, ** DO NOT IGNORE IT! *** *** IF THIS HAPPENS, DO NOT ERASE OR FLASH ANYTHING UNTIL YOU ARE SURE WHAT ELSE HAS BEEN DETECTED. *** ONLY CONTINUE WHEN YOU ARE SURE THAT YOU AREN’T GOING TO ACCIDENTALLY FLASH SOME LSI CONTROLLER YOU DIDN’T REALISE WAS THERE. “sas2flash -c CARD_ID -list”* will show each card in turn. Use this to check what you’re seeing, and take care to remove (or avoid modifying) any LSI controllers except those you mean to crossflash. After you’ve noted down or photo’ed any info you want to, reboot.

  2. Reboot into MSDOS, and using Megarec.exe, backup the existing SBR data (although you probably won’t ever need it) and then erase the card and all NVRAM data, including writing empty SBR data to the card. I’m not sure how important erasing the SBR data is (I never needed it!) but it can’t hurt and Megarec allows the original to be rewritten from your backup at a pinch. There’s a blog post linked at the end of this post, about it, and Google may help. It seems to be relevant to drive detection in a few (rare?) cases. But flashing a standard default “empty” SBR seems to work absolutely fine for virtually everyone, judging by how rarely it comes up in posts. The important step is to be sure you have truly erased the card properly. This is crucial or some of the flashing steps below will probably fail when you get to them.* I want to underline that last bit - most versions of sas2flash either did not properly erase the card for me, or hit issues trying to erase it. Basically it looks like sas2flash doesn’t always play well with flashing other manufacturer’s cards until they had first been erased using Megarec. Sometimes steps below seemed to work but didn’t, sometimes it point-blank refused to flash. Some versions probably do work. As I’ve said above, I found it best to use megarec only, and take the guesswork out of it.* The 3 commands to run are: megarec -readsbr CARD_ID FILENAME backup the card’s sbr data to your USB stick “in case”You probably wont need it but if you do, you can restore it from USB or hard drive at any time with the same program (the command is “Megarec -writesbr CARD_ID FILENAME”). * megarec -writesbr CARD_ID sbrempty.bin writes blank sbr data to the card. sbrempty.bin is a standard file that’s often included when you download Megarec. I’ve included it in the attached zip file so this command will work. *If this or the previous action fails, it’s probably fine according to guides online, but for me it didn’t give an error. megarec -cleanflash CARD_ID erase the card completely
  3. Reboot again.From this point on we’re done with megarec and we only need to reboot into MSDOS/EFI (whichever sas2flash requires). That is, unless we need to go back to 2 and wipe the card to start again :), or have a rare card that needs its sbr flashed for any reason (I’ve never had that). Note: From here on until we’re done, sas2flash -list will give incomplete or inconsistent output, such as “SAS ID undetermined”, or “card model = Dell but firmware = LSI”. That’s fine because we’re upgrading the card in stages and it doesn’t yet have that info completely or at all.

  4. Update DELL H200 IR to DELL H200 IT (or the IT firmware version for whatever manufacturer your card is). Use LSI’s **P5 EFI** flasher for this step. If your board doesn’t have EFI, then the P5 MSDOS flasher should work and I’ve included that in the zip file as well. The command is: sas2flash -c CARD_ID -o -f FILENAME ** The reason we use the P5 flasher here and in the next step is that the newer flashers are configured not to work with anything except LSI-branded cards and firmware, so until a few more steps are done, this is the flasher we have to use. Once we’ve got the card off Dell and onto LSI IT firmware (of any version, however old) then we can switch to using the latest LSI flasher without problems. I found no other version of the flasher that worked reliably, having tried LSI’s P7 EFI, a DELL flasher, etc. That was the one that worked. For Dell variants of the 9211-8i, the A10 9211-8e firmware is known to work well as an IT firmware to flash in this step, as explained and linked above. The filename within the download is 6GBPSAS.FW. Confusingly, Dell also use the same filename for their IR firmware, you have to check the download package for which it is. Using any other flasher than the P5 flasher, whatever brand the card was (whether Dell or LSI), gave the error “Cannot flash IT firmware over IR firmware”, either at this step or if this step worked, then when crossflashing to LSI P7 at step 5. However when I used the P5 flasher, even if I had got this message and even if “IR” appeared in the sas2flash “-list” printout, the P5 EFI flasher still crossflashed the IT firmware correctly (as verified after reboot). However if the IR->IT flash screws up badly, or just can’t write, go back to step #2 **cleanflash CARD_ID, wipe the card again, and redo this step until it works. You need the card to be running your own-brand IT (not IR) firmware first of all, in order to go anywhere else, including switching to LSI IT. The error “Cannot flash IT firmware over IR firmware” at a future step means that you didn’t get this step to work correctly. Either you thought you were flashing IT but you actually weren’t, or the firmware flash process didn’t work when you thought it had. In either case the solution is the same - go back to step 2, erase again, and double check the firmware you tried to flash really is IT and is one of the earlier firmware versions up to say 2009-2013 so there’s no doubt it can be flashed using the early flashers such as P5 (this isn’t usually an issue but I’m mentioning it “just in case” and in case future updates break compatibility). If in doubt, ask for help finding “known-good” IT firmware for your brand. It will be one of those two things only. (Note: In theory you could probably flash these two steps the other way round, meaning, crossflashing Dell/Fujitsu IR to LSI IR P7 first, and then LSI IR P7 to LSI IT, but I haven’t tried that, nor have I found any post saying that someone else has tried it. If someone does try it, I’d like to hear if it worked. Post a comment if you do!) The flashing process may give an error at the end, “due to errors, remaining commands were not executed”. Provided the flash itself says it successfully completed, this can be ignored.

  5. Reboot (to ensure all’s clean) and check what the card data now says: sas2flash -c CARD_ID -list For my H200’s, it showed Dell 0x2713 + 6GBPSAS, and not 0x2713 (IR). So far so good.

  6. Crossflash Dell A10 IT to LSI P7 IT (usually called 2118it.bin) sas2flash -c CARD_ID -o -f FILENAME I used the P5 EFI flasher + LSI P7 IT firmware (“2118it.bin”), which did what was expected and asked if I wanted to change the vendor ID (obviously “Yes”). If it doesn’t do that, then something’s not right. Check you included the “-o” option in the command, it won’t ask if you don’t. * This is the other step that most often falls over. It’s very dependent on exactly the right flasher and firmware versions. LSI’s P5 flasher + LSI P7 IT FW works well in combination.* I tried dozens of combinations and downloads, that seemed the reliable one. Other combinations of flasher and firmware gave errors such as: * “Failed to validate MfgPage2” which really means “Crossflash attempt blocked because your card isn’t bought from LSI so we’re going to be annoying” (the Vendor IDs in NVRAM and new firmware don’t match) - this is why we’re using a pre-P7 flasher for the crossflashing step, as the earlier flashers were as reliable but allowed crossflashing. A few minutes with the P20 flasher and a debugger would also probably fix this (see SAS2HAX link at the very bottom). or: * “Cannot flash IT firmware over IR firmware” which really means step 3 didn’t actually work or erasing at step 2 didn’t work, or the card’s checksums got screwed by the flasher and you need to start again by rewiping from step 2. Error messages after the flash succeeds, such as “due to errors, remaining commands were not executed” don’t seem to matter provided it looks like the flash itself succeeded. If you get an unexpected error, check first of all that you didn’t leave out the “-o” in the command, it’s easy to do. Without that, it won’t ask if you want to change vendor ID, even if you’re using the right versions. (Note: See also the sas2flash analysis and modified efi flasher “SAS2HAX” which skips the manufacturer ID check, at the end of this page. However as the normal unhacked LSI P5 works fine, I suggest sticking with that for certainty if new to this.)
  7. Reboot, and just check you’re on LSI IT firmware (see below), and if it’s worked, then in theory it’s all over except for a last update to P20, a SAS ID and a little tidying up. * From here on, use the P20 not the P5 version of SAS2flash, unless you have to go back to previous steps. We only used the P5 version to allow crossflashing to LSI and to allow flashing of non-LSI firmware, if that succeeded then we don’t need it any more.* Use sas2flash -c CARD_ID -list to confirm we’re now showing LSI details for the card. Should show “LSI” not Dell, and “(IT)” after the 4 hex digits of the firmware product ID in the middle. If it doesn’t, you have some troubleshooting ahead of you. Hopefully it did work. If it does show LSI IT firmware as expected, then there isn’t much else to do and you shouldn’t have any problems. All that’s left is: * Update the firmware again (also called “2118it.bin”), this time from LSI P7 to LSI P20 or other latest LSI firmware. Use the standard LSI P20 (or latest) sas2flash flasher and firmware, it’ll be fine now. The flashing command is the same (sas2flash -c CARD_ID -o -f FILENAME). If you need a specific version (P16, P20) for compatibility with your system, choose that version. * Set a SAS ID (“sas2flash -c CARDID -o -sasadd HEXSTRING). Some guides suggest starting with some specific first 8 hex digits, or at least keeping the initial digit as “5”. I didn’t have a reason not to, so I kept the first 8 digits as they had ended up after wiping and reflashing, and just modified the last 8 hex digits shown in “-list” to something unique for my system, in case any of the first 8 were LSI specific or expected. Technically the WWN should be unique worldwide, as with MAC addresses, but in practice that’s probably a non-issue for most of us here. The command takes a single string of 16 hex digits, so do not include any hyphens, underscores, spaces or punctuation, even if the original SAS ID included them. * Consider whether to flash a BIOS boot rom (a.k.a. option rom or orom), and whether to also flash an EFI boot rom. Steps and usual filenames are above, just use the -b option (sas2flash -c CARD_ID -o -b FILENAME). You can flash both, one, or none of these boot roms. You don’t need to flash any boot rom for an HBA, and it slows down boot, so you might decide not to. Unless you really need/want these, it’s usually best not to flash a boot rom as it won’t do much except slow your system boot. I didn’t bother.
  8. Give it one last reboot to MSDOS/EFI for certainty and check with “sas2flash -c CARD_ID -list” that everything now looks as you expect. If you flashed either or both boot roms (bios or EFI) that’ll be listed as well. (The EFI boot rom may be described as “UEFI” or “BSD” but it appears on the next line after the bios boot rom).
  9. Done. Enjoy. Check it can find disks on all ports (if it finds some but not others then sbr might be a factor, it’s rare but has been reported, you’ll have to Google it or check the link below) and check that the card seems to allow FreeNAS to see a disk on any of its ports, properly. If there’s an issue with disk finding or operation, once on the right firmware, consider the cable first. Even if a SAS cable can detect SATA drives it still might not work properly for SAS - I’ve had this and it’s a hard one to trouble shoot if you wrongly assume that “my card can see SATA drives” implies that the cable is sure to be good for SAS as well. If in doubt, Adaptec/Microsemi make reliable cables. Dell’s adapter cables which came with my card also look fairly well-made. The cheap thin SAS cables off EBay that work for SATA but not SAS drives (no idea how that can be but it’s what happened to me, first time) could be a hidden issue. If the issue’s still not solved and it’s not the cable, and other troubleshooting fails, then SBR might be a candidate (see link below), or ask for help in the forums.

OTHER RESOURCES

OPEN QUESTIONS

  1. Can the P5 flasher, flash recent (P19-20) LSI firmwares? In other words, can you use the LSI P5 flasher to jump in a single step from (say) Dell P7 IT to LSI P20 IT, or do you have to go via the LSI P7 IT? (Credit to IceBoosteR for this question: LINK**) **
  2. Every guide discusses going (for example) Dell IR -> Dell IT -> LSI IT (with update if needed). Can it be done the other way as well: Dell IR -> LSI IR -> LSI IT?

Updated: