Nvidia CUDA & ATI Stream (CAL) FAQ

Warning! Please read the whole FAQ first, to make sure you have all the details. Do follow the links (those blue lines) to other FAQs or to external sites for drivers and things like that. That way you don't have to ask in forums where particular parts go, such as that elusive cc_config.xml file.

Since a few people missed where the elusive cc_config.xml file comes from and where it goes, I've rewritten this FAQ to start with that information. The next person still missing the information is a dork. :-)

Using a cc_config.xml file. In a lot of the help below I am pointing out you can use particular lines in the core client configuration, or cc_config.xml file.

By default BOINC does not come with a cc_config.xml file. You will have to make one yourself. It resides in your BOINC Data directory, whose path you can find in the BOINC start-up messages. This directory may be hidden.

• See the core client configuration FAQ for help on how to set up the cc_config.xml file.

• See the Big BOINC 6 FAQ for the default directory paths and help on how you can set up Windows so you can see hidden files and folders.

What is CUDA or CAL (Brook+)? CUDA and CAL (Brook+), the next best thing since sliced bread... or so they want you to believe.

Both these versions are a method of doing calculations with the use of your videocard's GPU, or Graphics Processing Unit. CUDA is used by Nvidia GPUs only, while CAL is used by ATI/AMD GPUs only.

Inside the GPU there are multiple stream- or shader processors. While these are usually used to render 3D graphics in detail onto your monitor, they can also be used for doing other sorts of calculations.

That's what is being used on some of the projects running under BOINC. Using CUDA & CAL calculations will go much faster than on your CPU, since all the stream processors inside the GPU will attack the task at the same time, running all the calculations in parallel. This in contrast with how they normally run on your CPU, where one task runs on one CPU or core only, multi-threaded projects excluded.

While running work on your CPU(s) can already slow your computer down, imagine what will happen when you also run work on your GPU. It's already in use by your system to show you everything that happens on your screen, thus adding a load of work onto that, it can cause severe stability problems.

While different GPUs can be mixed used in computers, in general the work you get for projects cannot be exchanged between GPUs. Most projects use CUDA only, which cannot be done on ATI GPUs. Some projects use CAL only, which cannot be done on Nvidia GPUs.

I will be handling some of the problems you can come across, and possible solutions for them, in this FAQ.

Projects with CUDA Albert@Home (Einstein test project) ** GPUGrid Seti Beta * Seti@Home * Einstein@Home ** Milkyway@Home ** Collatz Conjecture ** Moo Wrapper **

Projects with ATI Stream (CAL) Albert@Home (Einstein test project) ** Collatz Conjecture ** Milkyway@Home ** Moo Wrapper

• Windows only, Linux and Macintosh require 3rd party applications or are not available.
  • Macintosh GPU apps available.

Absolute Minimum System Requirements 1. You will need at minimum BOINC 6.4.5 (or newer) (Windows, Linux and Macintosh).

1a. For ATI (CAL) support you need the 6.10 versions of BOINC. Get BOINC 6.10.17 from the download page.

2. You will need a CUDA capable nVidia video card with at least 256MB video RAM. For a list of CUDA capable GPUs, look at this list of cards.

2a. You will need a CAL capable ATI video card with at least 256MB video RAM. Compatible GPUs are the HD2xxx, 3xxx, 4xxx and above, or the ones on this list.

3. For 6.4.5 to 6.6.41 you will need at minimum nVidia CUDA 2.0 (177.35 and above) video card drivers.

3a. For 6.10 and later you will need at minimum nVidia CUDA 2.2 (185.85 drivers and above), but your best bet is to use the latest drivers available. Note: The minimum memory requirement on the GPUs goes up to 384MB of video RAM from these drivers onwards.

If your present drivers are below this version, update first. If your present drivers are below this and they're the last available for your hardware, you cannot update to 6.10; stay at the last 6.6.x version for your OS.

3b. For applications built with CAL 1.2 a minimum of Catalysts 8.12 are needed For applications built with CAL 1.4 a minimum of Catalysts 9.3 are needed. Check at the project (forums) which CAL version their applications are before installing these drivers!

4. When you are using a laptop and the above Nvidia drivers do not work on your system, plus the manufacturer of your laptop has not updated its drivers, you can first check Nvidia's laptop drivers to see if those work with your system. If they do not, then check Laptopvideo2Go, which seems better prepared.

5. If you are using a Macintosh computer, you need at minimum CUDA 2.3 drivers and OS X 10.5.2 (Leopard). You can get your drivers from the CUDA download site, choose Mac OS from the drop-down menu.

When you meet the above requirements and are ready to go, go attach to any of the projects that use a CUDA application at the moment. When you have done so, the application will download automatically.

My system slows down when I run CUDA or CAL. I can't work like this! When your system runs CUDA or CAL tasks, the GPU is saturated with whatever task it is doing. A large portion of your video RAM is being used by this task as well. So of course things will slow down.

The GPU draws everything that you see on your monitor. If at the time it is busy with a lot of calculations on all its internal processors, it won't have time to draw the screen. 3D gaming will also try to saturate the GPU, so you cannot have both CUDA or CAL and the game occupy the same space at the same time.

Seti has at this time tasks that have a very low angle range (VLAR) which will slow down when they are being calculated on the GPU. This slowdown will also have its effect on your computer. A fix for this is forthcoming in a next version of the science application.

The official advice is: - If you have a dedicated gaming rig, then do not run BOINC/CUDA/CAL at the same time you are playing your game, or only run it when you aren't gaming. - When you love to watch DVDs or use a TV tuner, do not use BOINC/CUDA/CAL at the same time. This will impact your viewing experience. (stutters, slow-downs) - When using a screen saver, remember that the GPU is used to render the screen saver's graphics. Just do not use a screen saver and turn off your monitor before you leave, or let it go to blank.

Ways around it? First and foremost is to exit BOINC before you start gaming or doing your other GPU intensive thing. BOINC Manager->File->Exit, check "stop running of science applications when exiting BOINC Manager" and click OK. Restarting BOINC Manager will also restart the BOINC service (if you installed it that way).

Second, you can suspend BOINC before you start gaming or your other GPU intensive thing. (*) BOINC Manager->Advanced view->Activity->Suspend. To resume follow the same path and set it back to Run based on preferences.

Third, you can have a cc_config.xml in your BOINC Data directory that will suspend your BOINC (under Windows only, I'm afraid) automatically when any of the exclusive applications you told it to look out for is detected in main memory. (*)

To do so, set up the cc_config.xml file with the following options (example of my cc_config.xml): <cc_config>

 <options>
    <exclusive_app>Speed.exe</exclusive_app>
    <exclusive_app>Crysis.exe</exclusive_app>
    <exclusive_app>FarCry.exe</exclusive_app>
    <exclusive_app>NFSHP2.exe</exclusive_app>
    <exclusive_app>FarCry2.exe</exclusive_app>
    <exclusive_app>Fallout3.exe</exclusive_app>
    <exclusive_app>RA3.exe</exclusive_app>
 </options>

</cc_config>

Since BOINC 6.10.19, there is also the <exclusive_gpu_app> option, used in the same way as the <exclusive_app> option. The difference is that BOINC will then only suspend work being done on the GPU, for as long as the program defined is in memory. It will continue to do work on the CPUs.

<cc_config>

 <options>
    <exclusive_gpu_app>Speed.exe</exclusive_gpu_app>
    <exclusive_gpu_app>Crysis.exe</exclusive_gpu_app>
    <exclusive_gpu_app>FarCry2.exe</exclusive_gpu_app>
 </options>

</cc_config>

You can also combine the two options.

<cc_config>

 <options>
    <exclusive_gpu_app>FarCry.exe</exclusive_gpu_app>
    <exclusive_app>Speed.exe</exclusive_app>
 </options>

</cc_config>

You only need to tell what application name it needs to look out for, no need for the full path to the name. If you already have one, you can add the <options></options> part to your existing cc_config.xml file.

Watch out with launcher files. For example, Fallout 3 uses the Fallout 3 Launcher, which does not stay in memory after you launched the game. So do not use the launcher file as the exclusive app, but the game executable's name instead. I have tested this with the Oblivion launcher as well, here it is the other way around. If you add the Oblivion executable only, BOINC will not suspend. You will have to use the Oblivion Launcher name.

(*) There is a problem when you have your preferences set to Leave applications in memory while suspended. Any task still in the video card's RAM will stay there until you resume BOINC. Although it will not use your GPU, the amount of memory that a Seti CUDA tasks takes up (200MB+ of the VRAM) may still have impact on your other activities.

Note: BOINC 6.6 will unload the task from video memory when you suspend BOINC and the task has at least checkpointed once. Since BOINC 6.10.32, doing benchmarks will leave all applications in memory, this includes what's in the GPU's memory.

Can't I turn the damn thing off? Yes, you can. E.g. in Seti, go to your Project preferences, edit them, take the check mark off of "Use NVIDIA GPU", save the changes. Then open BOINC Manager->Advanced view->Projects tab->select Seti->click Update. (Of course if you're using the GPU on another project, change the preferences on and update BOINC for that project).

Some projects have a separate "Use ATI GPU" preference. Normally those with official ATI applications.

Any work that is still appointed to the GPU will still have to be done or you have to manually abort it. In the case of Seti CUDA work, this is all work appointed to the 6.08 application (BOINC Manager->Advanced view->Tasks tab, Application column). You will not receive new CUDA work, though.

Other projects have these preferences as well.

In BOINC 6.6 (and above) you have the option to deselect the GPU through your cc_config.xml file. In this case add <no_gpus>1</no_gpus> to the <options> section of the cc_config.xml file. You must exit and restart BOINC for this option to be of effect. A simple re-read of the config file won't disable the GPU.

<cc_config>

   <options>
       <no_gpus>1</no_gpus>
   </options>

</cc_config>

With BOINC 6.6.7 and above you can also go to your Seti computing preferences and state that you only want to use the GPU when the system is idle. This is the best situation to use CUDA in.

From BOINC 6.10.19 onwards, the client also has the --no_gpus command line switch. That way you can start the BOINC client in the following manner:

boinc.exe --no_gpus --detach

The extra line of --detach closes the command line window, but leaves the client running.

Seti advises that you only run CUDA when you are not doing anything else with the system.

Yesterday I used CUDA or CAL, but today when I start up BOINC it does not detect any GPUs/co-processors. When will you fix that? This is not a BOINC problem, but something between the videocard drivers and Windows going wrong. It happens mostly on Vista and Windows 7 systems, where the driver does something illegal and the OS automatically uninstalls the driver, then reinstalls the previous OS supplied driver.

Although previous versions of BOINC would throw away any work in cache for the GPU, the latest version of BOINC (6.10.17 or above) will retain the work, until you either reinstall the drivers, or until the work times out by deadline.

I am using Remote Desktop to log in on my home-machine, but then BOINC does not detect a GPU. Can't you fix that? This is not a BOINC problem. When using the Remote Desktop Procedure (RDP), you are using the drivers that come with the RDP. These only allow the RDP, not anything else fancy, so if you want that fixed, you'll have to yell at Microsoft. Good luck. :-)

BOINC 6.10.17 and above will stop work being done on the GPU when BOINC detects someone logging in through RDP. It will then put this work on hold until you log out of the system.

Where did the screen saver for the projects go to? All I get is the BOINC logo! The screen saver uses the GPU to draw the OpenGL graphics on your monitor. CUDA is already using the GPU. When they would allow a screen saver at this moment, you have enormous slow downs and a higher possibility of errors. Which is why the Seti CUDA application does not come with a screen saver executable at this time. This may change in the future.

Although it has always been possible to run any screen saver together with CUDA or CAL, it is better to do this on videocards that have enough memory -- 1024MB and more.

Not all projects allow a screen saver program together with their CUDA application. The only one I know of that does so is the Einstein project. Ask other projects.

I have 4 GPUs and want to crunch on them only, leaving my CPUs free. Isn't that possible? Yes, that is also possible, but only with BOINC 6.6 to BOINC 6.10.33 (*). In this case, add <ncpus>0</ncpus> and <use_all_gpus>1</use_all_gpus> to the <options> section of your cc_config.xml file. Let BOINC re-read your config file and the CPUs are off and all GPUs are used. Do know that for the moment, one CPU will always be used to supply the GPU(s) with work. This is minimalistic of impact, though. About 4% per GPU and not on a constant basis. Only when tasks are swapped from disk to VRAM and back, will the CPU be used. At all other times the GPU is used.

<cc_config>

   <options>
       <ncpus>0</ncpus>
       <use_all_gpus>1</use_all_gpus>
   </options>

</cc_config>

A new option appeared in the Seti Project preferences, with which you can disable your CPU(s). For this select:

Use Graphics Processing Unit (GPU) if available ON Use Central Processing Unit (CPU) OFF

(*) Since BOINC 6.10.34, <ncpus>0</ncpus> again means that you disabled the <ncpus> setting. It no longer means you want to use zero CPUs, since this is counter-intuitive as all GPU applications use at least one CPU.

I have 2 GPUs in SLI, BOINC recognizes both but CUDA or CAL only crunches on one. What is up with that? When you are using SLI, all the GPUs you have are seen as one GPU, producing one single output.

For CUDA: If you want to use the GPUs independently, you will need to disable SLI in the Nvidia Control Panel, then restart BOINC. The bridge between video cards can stay on.

Note: Since driver version 190.38, BOINC will recognize the GPUs as separate ones even if you are running in SLI mode.

For CAL: If you want to use the GPUs independently, you will need to disable SLI in the Catalyst Control Panel, then restart BOINC. The bridge between video cards can stay on.

OK, so now I disabled SLI and BOINC only recognizes one video card. What is up with that then? Some people report that in order to have both video cards recognized and be able to use them, you can try to extend your desktop over all your videocards; if that doesn't work you need to have a monitor attached to each video card, or add a VGA dummy (requires soldering) to the VGA outlet of the other video card(s). This how-to needs no soldering.

Also set your cc_config.xml file to use the <use_all_gpus> option.

<cc_config>

   <options>
       <use_all_gpus>1</use_all_gpus>
   </options>

</cc_config>

How do I use only one of my multiple GPUs? You can tell BOINC which GPU to ignore using the <ignore_cuda_dev>0|1|2|3</ignore_cuda_dev> and <ignore_ati_dev>0|1|2|3</ignore_ati_dev> flags in the <options> section of cc_config.xml

GPUs are counted per brand and from the first in your system to the last, while the first GPU of the same brand is designated GPU 0, the second GPU 1 etc.

So for example, you have a 3 GPU system, two Nvidia and one ATI. You only want BOINC to use the first Nvidia and the ATI, not the second Nvidia.

<cc_config>

   <options>
      <ignore_cuda_dev>1</ignore_cuda_dev>
   </options>

</cc_config>

The above code will take care of that. Put that cc_config.xml in your BOINC Data directory (or adapt it into an existing cc_config.xml file), then exit and restart BOINC.

If you want to ignore both the second Nvidia and the ATI, you use the following:

<cc_config>

   <options>
      <ignore_cuda_dev>1</ignore_cuda_dev>
      <ignore_ati_dev>0</ignore_ati_dev>
   </options>

</cc_config>

In BOINC Manager it shows that the CPU time for a CUDA task is mere seconds, but the task runs for hours. Why is this? BOINC does not follow GPU time yet. That's planned for a future version. For now it will only count CPU time. For CUDA and CAL this means the time it takes the CPU to transfer the task from disk into the video RAM and when the task is done, transferring it back.

The GPU will do the whole calculating of the CUDA or CAL task.

I have made my own BOINC Manager, based on BOINC 6.6, which shows both the CPU Time and the Wall-clock time columns in the Tasks tab. It's for Windows only and works with BOINC versions as far down as 6.4.5 (so on all versions capable of CUDA). You can download it in 7-Zip format from my Microsoft Skydrive.

Just unzip/unRAR to your BOINC directory and run it as is. Since it's called boincmgr76.exe it won't overwrite your original BOINC Manager. I am not supplying boinc.dll with it so not to break your existing BOINC version, but for the older versions of BOINC 6 it needs the sqlite3.dll file that's included.

A newer and updated version, built against the latest available 6.10 GUI is now also available, from here (zip compressed) and here (7-zip compressed).

A Linux version of the same file is available from Kathryn's Skydrive. Her comments:

"I've built the 32 bit Linux version of this. It's simply called boincmgr. I have no idea if it'll run on anything but my machine. I've never built binaries for distribution. It was compiled with the --enable-unicode flag for wxWidgets. My suggestion is if you want to try it, download it, check the md5sum (4fa7a16a56dadb2e26a61a81b97591ca) and then run ldd on it.

When I use CUDA or CAL, everything slows down. In Task Manager I see Explorer.exe is using up a lot of CPU cycles. Why is it doing that? We don't know. Other than the normal slow-down caused by you using the GPU for other things than drawing the desktop, this is neither BOINC, nor the science application that is causing explorer.exe to take up CPU cycles. It is more likely that it is something local on your system.

To help out the people that are trying to help you, run FileMon and Process Explorer to see what is happening on your system. Post logs from those programs.

For some people a simple reboot made it go away.

I got a report in that updating to the CUDA 191.07 drivers may fix this problem, although this person also updated his motherboard chipset drivers. So if in doubt, try either one.

When I ran BOINC in XP as a service, it would detect the GPU. Now I have Windows Vista or Windows 7, when BOINC is installed as a service it won't detect the GPU. When will you fix that? It is not something for BOINC to fix. This is a Microsoft security feature, where it stops your user account from running in the same session as your drivers and services are run from.

If it has to be fixed, it will have to come from Microsoft, or the GPU driver manufacturers have to find a way to run their drivers without using them as a service installation.

There's a document on this at Session 0 isolation (Word document, can be loaded in Open Office Writer, not in Wordpad).

An excerpt: In Microsoft® Windows® XP, Microsoft Windows Server™ 2003, and earlier versions of the Windows operating system, all services run in the same session as the first user who logs on to the console. This session is called Session 0. Running services and user applications together in Session 0 poses a security risk because services run at elevated privilege and therefore are targets for malicious agents who are looking for a way to elevate their own privilege level.

The Microsoft Windows Vista™ operating system mitigates this security risk by isolating services in Session 0 and making Session 0 noninteractive. In Windows Vista, only system processes and services run in Session 0. The first user logs on to Session 1, and subsequent users log on to subsequent sessions. This means that services never run in the same session as users’ applications and are therefore protected from attacks that originate in application code.

What Is Affected Any applications or drivers that are installed as a service are affected by the following implications. Some drivers are loaded within operating system services or processes that are running in Session 0, and those drivers are also affected by the implications of the Session 0 changes.

What about the Macintosh and Linux versions, where are they? CUDA finally works correctly with Mac OS 10.6.3

But as ever, if you upgrade to OS 10.6.3, 2 issues may arise:

• The upgrade may delete your login item for BOINC Manager. If this happens, BOINC will not launch automatically when you log in or restart your Mac. Just run the BOINC installer again to fix this.
• If you are running CUDA tasks, you must upgrade to version 3.0 of the CUDA drivers, which you can download here.

As far as the Linux version is concerned, BOINC 6.10.17 has CUDA and CAL support. If you run into problems and your GPU is one of the newer ones, your best run CUDA with the 185.85 drivers. I have had reports that this driver doesn't work with the Nvidia GTX 280 and above versions, though. You may want to check what Nvidia says about that.

When will ATI GPUs (or any other non-Nvidia brand GPUs) be detected? BOINC 6.10.17 and above have ATI detection support, though this doesn't say that projects will then immediately have applications that run on your card. The biggest problem with ATI at the moment is that they do not give any support on porting over project's science applications to work with their Stream SDK. People will have to figure this out for themselves and many project administrators just do not have the time or money to learn a whole new programming interface (Brook).

However, not all is bad. On the Milkyway project one of the users has made an ATI application. It works independently of BOINC, so you do not need BOINC 6.4.5 to get it to work. Older versions will work as well. Check the threads there for more information, or go the the zslip thread for the download link.

The Collatz Conjecture project has also adopted the use of the ATI GPU.

Now really, when will BOINC support OpenCL? OpenCL support has been in BOINC ever since it started detecting GPUs. Further hardware support should come in the future, but this has to do with what drivers you install and the accompanying OpenCL.dll -- too technical to go into here.

For minimum system requirements look: - Here for ATI. For the OpenCL FAQ, see here.

I am continuously getting memory errors on tasks I run. A reboot didn't help. What should I do next? Test GPU memory, with either of these apps, or both:

MemtestG80. Copy the cudart.dll and cufft.dll files from you setiathome.berkeley.edu folder to the folder where you installed MemtestG80file. You will need the old DLLs from before CUDA 2.3, cufft.dll file size = 1148kB, cudart.dll file size = 275 kB.

Run it from a command-line window (Start->Run, type cmd, click OK. In the command line window use DOS commands to navigate to the correct directory

e.g. when installed in C:\cudatest cd\ {Enter} cd\cudatest {Enter} memtestg80 {Enter}

You need to enter the graphics card memory speed and GPU core speed. If you don't know what they are you can get and run GPU-Z to find out.

OCCT Perestroika 3.0.1, choose the CUDA MemTest tool in the \bin\CUDAMemTest\ directory.

Post the results in your project forum of choice.

Original writer	Original FAQ	Date & time
Jorden	471	17-12-2008 22:07:51