Testing the videocard memory
When your videocard memory is broken it will cause all kinds of artifacts when just doing monitor stuff, but certainly when doing calculations on the GPU. You may want to run a memory tester on the videocard.
Video Memory stress Test
Video Memory stress Test is from back in 2008 and by Misha Cherkes, the same guy behind the useful Check Flash USB drive testing tool. This tool allows you to test your video card RAM using one of three different test types, DirectX, CUDA and OpenGL. It looks similar to the system memory tester Memtest86+ and works in the same way running a number of predefined patterns to test the memory as thoroughly as possible. There’s also an offline version that can be booted using CD or floppy.
Video Memory Stress Test
CD/floppy version of Video Memory stress Test
MemtestG80 will test nVidia graphics cards that are CUDA enabled, CUDA has been around since 2007 and classic old video cards such as the GeForce 9600GT, 8800GT and 8800GTX should be supported up to current day cards. MemtestCL is slightly different and supports OpenCL based cards, this covers a different range which includes AMD/ATI 4000 and 5000 series or above, and nVidia 8000/9000 series or newer.
Both Memtest versions are command line based. 
EVGA OC Scanner X
Although this application doesn’t physically scan the memory for errors like the previous tools, it does run a series of burn in type tests that will stress the video memory and likely show any errors that the graphic card’s RAM produces. OC Scanner X is by the video card manufacturer EVGA and will work on any graphics card, not just their own. The downside is you need to have a free account at EVGA.com to download the tool.
After downloading and installing OC Scanner X, run the program and select the 3D test and resolution you want and then make sure to check the “Artifact Scanner” box. Run the test for several minutes and any artifacts the program detects during the testing will be counted. Constant artifact counts over more than one run of the tests could mean issues with the graphic card’s memory.
OC Scanner X
Another burn in tool that has a video card memory error checking ability is an older version of OCCT, for some reason this function seems to not be available in newer versions or has been merged in with another test. It runs a series of Memtest like scans and the user sets the number of passes before starting the scan.
After running the program go to the GPU: MEMTEST tab and select the number of passes although the default of 10 should be quite thorough, dropping it to 5 would be enough for basic testing. Then press the On button.
Note: the memtest function of OCCT is based on nVidia CUDA and therefore will only work on nVidia graphic cards.