Xeon Phi, Xeon E5 and OpenCL under Ubuntu 11.10

Since the idbc_mic debugger still denies to work together with the Xeon Phi I did another experiment today and tried to get the Phi running with OpenCL. So I downloaded the latest version of the Intel OpenCL SDK 2013 and the drivers for processors and the one for Xeon Phi from: http://software.intel.com/en-us/vcsource/tools/opencl-sdk-xe. Like for the MPSS Intels software only supports RedHat and SuSe Enterprise. So I downloaded the three archives and unzipped them one by one with tar xvf. After that I executed in all of the three unzipped directories these both loops:

for f in *.rpm; sudo alien –scripts $f; done

and

for f in *.deb; sudo dpkg -i $f, done

to convert and install the packages for the two drivers and the general software. Within that process I didn’t get any errors and everything seems to be installed.

Since all linked linux sample projects from: http://software.intel.com/en-us/articles/intel-sdk-for-opencl-applications-xe-samples-getting-started/ lead to dead links, I had to take the Windows samples package from: http://software.intel.com/en-us/articles/intel-sdk-for-opencl-applications-samples-getting-started/ and to modify the MedianFilter project, so that it runs under Linux. All code, except the OpenCL kernel, is placed in one cpp file. The Makefile looks like this:

rm=/bin/rm -f
CC=gcc
DEFS=
PROGNAME= OCLTest
INCLUDES=-I. -I../common -I/usr/include/ -I/opt/intel/opencl/include
LIBS=-l OpenCL

DEFINES=$(INCLUDES) $(DEFS)
CFLAGS=-g $(DEFINES)

SRCS = ocl.cpp

OBJS = ocl.o

.c.o:
$(rm) $@
$(CC) $(CFLAGS) -c $*.cpp

all: $(PROGNAME)

$(PROGNAME) :
$(CC) $(CFLAGS) $(SRCS) -o $(PROGNAME) $(LIBS)

clean:
$(rm) $(OBJS) $(PROGNAME) core *~

But when trying to compile this, I get a warning that the numactl library wasn’t found on which the OpenCL libs are depended. A try to run the program anyway ended up in a crash because the needed *.so file couldn’t be found (no wonder :)). So I downloaded the numcatl package for Fedora 15 from: http://rpm.pbone.net/index.php3/stat/4/idpl/16839339/dir/fedora_15/com/numactl-2.0.7-1.fc15.x86_64.rpm.html, converted it to *.deb with the help of alien and installed it with dpgk. After that I found the Ubuntu solution from: http://packages.ubuntu.com/de/hardy/numactl. But it isn’t installed yet.

After that my program compiled and could be run. But the functions to find the OpenCL devices in the system only are aware of the Xeon E5 processors on the board and not on the Xeon Phi, although its driver is installed. I’m still searching a solution for this.

Update
My problem was not to install OpenCL but to initialize it correctly. So everything I explained here was correct and the installation was complete. First I used an example code from the Internet, to make sure, that the Phi is really registered as OpenCL device [1]. This created me the following output:

platform count: 1
device count: 2
1. Device: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
1.1 Hardware version: OpenCL 1.2 (Build 56860)
1.2 Software version: 1.2
1.3 OpenCL C version: OpenCL C 1.2 
1.4 Parallel compute units: 32
2. Device: Intel(R) Many Integrated Core Acceleration Card
2.1 Hardware version: OpenCL 1.2 
2.2 Software version: 1.2
2.3 OpenCL C version: OpenCL C 1.2 (Build 56860)
2.4 Parallel compute units: 236

At this point I could be sure, that the Phi is ready to work, but the next question was how to use it. After a while of reading the OpenCL Documentation I got a hint to the device type CL_DEVICE_TYPE_ACCELERATOR and with that I was able to calculate on Phi. The following code shows how I initialize my device (declarations and error handling ommited).

context = clCreateContextFromType(cprops,                                                                                                                                                                          
    CL_DEVICE_TYPE_ACCELERATOR,                                                                                                                                                                                
    NULL,                                                                                                                                                                                                      
    NULL,                                                                                                                                                                                                      
    &status);
 
status = clGetContextInfo(context,                                                                                                                                                                                         
    CL_CONTEXT_DEVICES,                                                                                                                                                      
    0,                                                                                                                                                                       
    NULL,                                                                                                                                                                    
    &deviceListSize);
 
devices = (cl_device_id *)malloc(deviceListSize);
 
commandQueue = clCreateCommandQueue(                                                                                                                                                                                       
    context,                                                                                                                                                                                
    devices[0],                                                                                                                                                                             
    CL_QUEUE_PROFILING_ENABLE,                                                                                                                                                              
    &status);

Now with device[0] I can do the rest of the initialization work.

 Sources:
[1] http://dhruba.name/2012/08/14/opencl-cookbook-listing-all-devices-and-their-critical-attributes/

 

Installing the new 2.1.5889-16 of MPSS on Ubuntu 11.10

Because of troubles with the debug software I updated my MPSS version with the hope, that this will resolve (some) problems. So I downloaded my the latest version from http://software.intel.com/en-us/articles/intel-manycore-platform-software-stack-mpss and followed my own tutorial from [1]. The only thing, that you have to change is the spec-file, if you want the Intel tools to show you the correct version of it. The appropriate file can be downloaded here: intel-mic-kmod-5889-16.

I did a complete uninstall of the previous version and after a restart I installed the newer one. Everything runs like before. Not worse, but still not better…

Sources:
http://www.theismus.de/HPCBlog/?p=1

Updating Phi’s bootloader and flash version

Since I still have trouble with Intels Debugging tools on Xeon Phi I decided to do an update day today. On march 21 Intel released a new driver version 2.1.5889-16 which I installed and will blog in a seperate post. After that I did the update of bootload and flash. To do that you have to no the stepping of your Xeon Phi. So you first have to execute /opt/intel/mic/bin/micinfo and look for the line Coprocessor Stepping. In my case this is B1. After that you can follow Intels description of the process given in [1], the readme of the MPSS package. I reprint it here in a shortened way:

This configuration is required for “SMC Firmware Version 1.7” or earlier. Execute /opt/intel/mic/bin/micinfo to identify the SMC firmware version installed on the card. 1) Set the coprocessor to a ‘ready’ state. user_prompt> sudo micctrl -r user_prompt> sudo micctrl -w user_prompt> mic0: ready 2) Update the SMC firmware only for installed B0 or B1 steppings of Intel(R) Xeon Phi(TM) hardware. Skip to step 3 if C0 stepping is present user_prompt>sudo /opt/intel/mic/bin/micflash \ -Update /opt/intel/mic/flash/<EXT_HP2_SMC_Bootloader_version> -device all Where <EXT_HP2_SMC_Bootloader_version> represents an SMC firmware file, which for this release is named EXT_HP2_SMC_Bootloader_1_8_4326.css_ab. The “ab” postfix means that the image applies to A and B steppings of the coprocessor. No reboot is necessary at this point. 3) The bootloader update is limited in functionality. The flash update must be run after the bootloader update for full card management availability (Section 7.3 “Intel(R) Xeon Phi(TM) Coprocessor Flash Update”). 7.3 Intel(R) Xeon Phi(TM) Coprocessor Flash Update. Notes: o Flash image files that end with .smc contain both the flash and smc firmware. o For in depth information about micflash options and tools check the micflash man pages. ******************************************************************************** * WARNING: Pay special attention when selecting the version of the Intel(R) * * Xeon Phi(TM) coprocessor flash image that will be flashed while performing * * the steps described in this section. Flashing the incorrect ROM file to an * * Intel(R) Xeon Phi(TM) coprocessor WILL cause the coprocessor to become * * unresponsive and will require it to be replaced. * ******************************************************************************** Steps: 1) Determine the flash image to install: Flash images are stored at the target directory, usually located at /opt/intel/mic/flash/. To choose the right flash image, refer to the following table that relates to each stepping of the Intel(R) Xeon Phi(TM) coprocessor: Stepping | Flash ROM Name ———-+—————————————— B0 | EXT_HP2_B0_0385-01.rom.smc B1 | EXT_HP2_B1_0385-01.rom.smc C0 | EXT_HP2_C0_0385-01.rom.smc 2) Set all coprocessors on ready state. user_prompt> sudo micctrl -r user_prompt> sudo micctrl -w mic0: ready 3) Install the flash image. In single coprocessor systems or multi-coprocessor systems with the same coprocessor stepping use: user_prompt> sudo /opt/intel/mic/bin/micflash -Update \ /opt/intel/mic/flash/ -device all In multi-coprocessor systems with different stepping values, update the flash for each coprocessor, specifying each card separately. For example, for B0 and C0 coprocessors installed in slots 0 and 1 use: user_prompt> sudo /opt/intel/mic/bin/micflash -Update \ /opt/intel/mic/flash/EXT_HP2_B0_0385-01.rom.smc -device 0 user_prompt> sudo /opt/intel/mic/bin/micflash -Update \ /opt/intel/mic/flash/EXT_HP2_C0_0385-01.rom.smc -device 1 4) Power cycle the host for the changes to take effect. Flash will be loaded after a host reboot. In a virtual environment, a reboot by micflash will only reboot the virtual machine.

Now micinfo shows me Driver Version: 5889-16, Flash Version: 2.1.01.0385, SMC Boot Loader Version: 1.8.4326 and uOS Version: 2.6.38.8-g9b2c036. This was the first thing, that I did with Phi, that completely runs like it is described in the documentation.

Since everything seems to be up to date now I will again try to get the debug tools running.

Sources:
[1] http://registrationcenter.intel.com/irc_nas/3047/readme-en.txt

Trying to get the Phi running under Ubuntu 12.04 LTS with kernel 3.5.0-26

As I reported in [1] I was able to get the Xeon Phi running on Ubuntu 11.10 with the 2.1.5889-14 mpss version and the latest available kernel for 11.10. After a complete crash of our cluster node we tried to use the chance to upgrade to 12.04 LTS with kernel 3.5 and I tried for two days to get the Phi running on this configuration.

The first thing when trying to follow my description in [1] is that the compilation in step 9

For compiling the kernel module go to /root/rpmbuild/SPECS. Here execute the command rpmbuild –bb intel-mic-kmod.spec. The sources are being compiled and a rpm-file is created. It is placed in /root/rpmbuild/RPMS/x86_64.

fails with several compile errors. This is caused due three changes in the linux kernel header files:

  1. The tty_driver structure has no field minor_num any more. It was removed because it was never used. The Intel software writes it at one position in linvcon.c but never reads this value. So I commented this writing out.
  2. The poll_table structure renamed her entry key to _key. So I changed this in all uses in the files miscif_api.c and miscif_nm.c.
  3. The SYSTEM_SUSPEND_DISK state was removed. It is only checked in micsif_nm.c and the only command that follows is a break. So I commented out those to lines.

So I extraced the driver code from the tar archiv to direction dirorg and copied these file to a new directory name dirpatch. There I applied the code changes described above. After that I created a second patch file with the diff command:
diff -uNr dirorg dirpatch > kernel35.patch
and added his execution/application to the spec file. I also changed the original spec file to remove the inherent error, that the name of the original first patch file there was wrong. The files are available in Kernel35 Patches. If you use this spec file to install the 2.1.5889-14 MPSS version on Ubuntu 11 as descripted in [1] you can skip step 7 from there and ignore the original patch (intel-mic-mpss21up1-kmodspecfile.patch) for the *.spec file.

With the kernel35.patch, the original intel-mic-mpss21up1-kmod-2.1.4982.patch  and the modified intel-mic-kmod.spec it was possible to compile the kmod kernel module and create the *.deb file following steps 8-10 from [1]. After finishing the installation process with this selfmade package, it is possible to get the status of the Phi via micctrl –status. It says ready and you can execute micctrl –initdefaults. The ssh keys are copied.

But then the disillusion. When trying to start the mpss daemon everything behaves normally and the Phi reports online. After 14 seconds (time taken from log file) the status changes to loss and the Phi tries to reboot until he reaches ready state. There he stays some time and after a while the hole system crashes. This behaviour is reproducible.

Nothing of my tries could change this situation. Warm or cold restarts or a complete new installation of the software (with repeating compiling). To uninstall the Intel stuff I generated a list with all installed packages with:
dpkg –get-selections > installed-software
and uninstalled one Intel package after the other with
dpkg -r PACKAGE
dpkg –purge PACKAGE

As last idea I tried to use the old Intel software version 2.1.4982-15. But with the same result in the end. As intermediate problem there was additionally a conflict of the intel-mic-gdb and the intel-mic-gpl package, where I had to force dpkg to install it, although they were writing to same positions.

After that I gave up the plan to use the new linux for the moment and went back to 11.10 with kernel revision 3.0.0-32, where I only needed about 30 minutes to get the Phi running, following my own step by step manual from [1].

Sources:
[1] http://www.theismus.de/HPCBlog/?p=1
[2] http://lkml.indiana.edu/hypermail/linux/kernel/1203.0/01593.html
[3] http://lkml.indiana.edu/hypermail/linux/kernel/1207.2/02974.html

Installing Intel Xeon Phi on Ubuntu 11

Since our cluster-node is running under Ubuntu 11.10, it was not that easy to get our Xeon Phi running in this system. That’s why we decided to give here a short how-to on this issue in the hope that it can help somebody else who is confronted with the same or similar problems. It is based on a descripition from http://www.poempelfox.de/blog/2013/03/06/#mic-xeonphi-mpss-with-ubuntu[1] but differs in some point. One reason is, that I used a newer version of the Intel driver. All steps were done as root, because some of the directories I needed where placed in the /root directory.

  1. First step is to download the driver from the Intel homepage: http://software.intel.com/mic-developer and go to “Tools&Downloads” and choice: Software Drivers: Intel® Manycore Platform Software Stack (Intel® MPSS). I used the version KNC_gold_update_2-2.1.5889-14 for SuSe SLES11 SP2 and you have to use it also on systems with newer kernels because it is designed for a 3.0xx kernel version.
  2. Unzip driver and MPSS via tar xvf KNC_gold_update_2-2.1.5889-14-suse-11.2.tar.
  3. For the next step you need the package alien. If it is not installed you have to install it via apt-get install alien.
  4. Go to the unzip directory at topmost level, where all nine *.rpm files are stored. Convert all packages to *.deb except intel_mic_kmod-2.1… because this package contains the driver kernel module, which needs to be compiled for your specific kernel manually. To convert all other packages use alien –scripts PACKAGENAME.deb. Alternatively you can use a for loop in your bash:
    for f in *.deb; do sudo alien –scripts $f; done
    to do the work for you. Before that you should delete the kmod rmp package. You won’t need it anymore.
    After this step you have eight *.deb packages in the directory where you have unzipped the files.
  5. Next step is the unpacking of the drivercode. Go to the subdirectory src. For this step you need the rpm-package. If it isen’t installed yet you have to do this with apt-get install first. Then unpack the files via rpm2cpio intel-mic-kmod-2.1.5889-14.suse.src.rpm | cpio -idmv. You will get to files. The first one with the driver data is called intel-mic-kmod-2.1.5889.tar.bz2 and the spec file intel-mic-kmod.spec.
  6. Now you will need the two files from zip archiv intel-mic-mpss-kmod-patch_usable. In this case you get a intel-mic-kmod.spec file for the compilation process of the kernel driver code and a  intel-mic-mpss21up1-kmod*.patch file which modifies the code before compilation to be compatible with the Ubuntu kernel.
    You can also download original versions from [1]. Then you have the two files mentioned above and additionally a intel-mic-mpss21up1-kmodspecfile.patch file. But then you have to do the changes to the *.spec file, which are described in the following step by hand.
    The files in the archive linked above are ready to use. So you can extract them and skip step 7. But you should read it to know, what was changed in the file.
  7. The original spec-file-patch can’t be automatically applied to intel-mic-kmod.spec because Intel changed the spec-file content for the 5889-14 release. So I had to integrate the changes by hand into intel-mic-kmod.spec. So I had to:
    – add the g in the line which starts with %define kernel abbrev. Like you can see it in the specfile patch.
    – add the line which starts with Patch1:. But here is an error in the original specfile patch. I had to fill in the correct name of the second file in the archive to the “Source0” line which is: intel-mic-mpss21up1-kmod-2.1.4982.patch.
    – comment out the line which begins with BuildRequires:.
    – add the %patch1 -p1 like you can see it in the patch-file.
  8. Now you have to move all the needed files in the correct directory to build the kernel module. So copy the edited spec-file to /root/rpmbuild/SPECS. The remaining (code) patch file and the intel-mic-kmod-2.1.5889.tar.bz2 must be placed in /root/rpmbuild/SOURCES.
  9. For compiling the kernel module go to /root/rpmbuild/SPECS. Here execute the command rpmbuild –bb intel-mic-kmod.spec. The sources are being compiled and a rpm-file is created. It is placed in /root/rpmbuild/RPMS/x86_64.
  10. Convert the created rpm file again to *.deb with using alien –scripts on it. Then copy the *.deb file to the directory where you unzipped the Intel stuff and where the other *.deb files are.
  11. Go to the Intel stuff folder and install all *.deb file with apt-get -f install. I did it in alphabetical order. Alternativly you can use a for loop in your bash:
    for f in *.deb; do sudo dpkg -i $f; done
    to do the work for you.
  12. Since the control tools for the Phi are placed in /usr/lib64 by default, you have to tell Ubuntu, that it should look there for the binaries. Create a new file named mic.conf in /etc/ld.so.conf.d and write the line
    usr/lib64 into it. Now execute the ldconfig command after that. [1] claims, that you can now use the Intel tools, like micctrl.
    One pitfall I (and another person [2]) had, when the systems tells that the libscif.so.0 is missing, was the fact, that there are two versions of this file on your harddisk. One as it is stated in [2] at opt/intel/mic/filesystem/base/lib64/ and another one in /usr/lib64/. The first one is a version which is compiled for the Linux version that is running on the Phi and not of use for miccntrl. So you have to add the /usr/lib64/ path (for example like it is described in [3]. After that micctrl –status should work.
  13. In the next step you have to change the init script for the mpss services, so that it works under Ubuntu. So start editing the file /etc/init.d/mpss. Just as desribed in [1] replace the line that reads startproc -t 1 $exec with these two:
    [ -d "/var/lock/subsys" ] || mkdir /var/lock/subsys
    start-stop-daemon --start --exec $exec 
  14. This passage can be exactly done, as [1] states. Open the file /etc/network/interfaces for edit an append:
    iface mic0 inet static
      address 172.31.1.254
      netmask 255.255.255.0
    to it. Save and close. For the network configuration I had also to run ifup mic0.
  15. Now everything should be prepared to start the Phi. mpss service. For that type service mpss start. The message Starting mpss stack: will appear. This can last some time, so wait and don’t press any keys. After a while you will get some missing rc_* messages but get the control of the console back. First test if the kernel module is loaded by typing lsmod and looking for a module named mic. After that type micctrl –initdefaults and micctrl –status. This call shall return you that the Xeon Phi is online now and ready to use.

Sources:
[1] http://www.poempelfox.de/blog/2013/03/06/#mic-xeonphi-mpss-with-ubuntu
[2] http://software.intel.com/en-us/forums/topic/366006
[3] http://blog.andrewbeacock.com/2007/10/how-to-add-shared-libraries-to-linuxs.html