How to set a custom shader in DirectXTK?

Jun 4, 2013 at 10:41 AM
I want to load an effect composed of a vertex shader, a geometry shader, and a pixel shader for rendering a 3D mesh which is in sdkmesh format. Although I have looked at the source code, there is no example about how to do it in DirectX tool kit. Anybody has experienced this problem, please help me. Thank you.
Coordinator
Jun 4, 2013 at 11:49 PM
Edited Jun 5, 2013 at 12:01 AM
By "effect", do you mean a Direct3D 11 FX .file with techniques for use with the Effects 11 system? The DirectXTK library loads models by doing a simple mapping to the built-in BasicEffects. While we have a work-item to support the Visual Studio DGSL pixel-shader authoring model, a generalized replacement for the FX system is a bit out of scope (mostly because FX is deprecated, the HLSL runtime shader reflection support isn't available for Windows Store apps, and the future of the fx_5_0 profile compilation is unclear). The SDKMESH format's material definition doesn't actually include a .fx file name, and the samples that use SDKMESH with Effects are done programmatically rather than through the file format.

If what you mean is that you have your own custom shaders set that you want to use as an override for a loaded Model, then you'll want to create your own IEffect interface object for it and you can use a custom EffectsFactory to bind it at load time. Alternatively, you can use the 'custom drawing' functionality.
Jun 5, 2013 at 2:57 AM
Edited Jun 5, 2013 at 3:00 AM
Thank you the reply. I do not mean fx file, I mean a hlsl file

In this topic, SetCustomShader, they load a compile shader, then apply to the batch rendering. Now I just want to load a hlsl file and apply to the 3D mesh rendering.
Coordinator
Jun 5, 2013 at 4:00 AM
You don't really 'load' an HLSL file, as much as you compile it into a 'shader blob'. You can either compile it at build time (which is recommended and easy to do if you are writing a C++ application using Visual Studio 2012), or you can compile it at runtime using the D3DCompile APIs as long as you are talking about Win32 desktop apps or just a test Windows Store app/Windows phone app that you won't actually try to submit to the Store.

See this blog entry for some background on HLSL compilation. For some details about profiles and how they relate to Feature Levels, read this one as well.

Once you have have the shader blob (from a binary disk file, returned from the D3DCompile API, or as a C++ header static array like we use for the BasicEfffect built-in shaderS), you create a Direct3D shader object from it. From there, it's a matter of binding it at draw time. There is the additional complexity that your vertex shader is needed to create an input layout. I cover this 'advanced drawing' scenario in the DirectXTK docs, but I assume you are using a DirectXTK Effect instance somewhere in all this (either a built-in one or a custom one).

You can try to do the same kind of 'custom state' override as that other thread shows easily enough as it's passed down through the Draw hierarchy and takes place after the 'effect' is applied. Again, the main issue is making sure your vertex shader object matches the input layout for the mesh.
Marked as answer by walbourn on 1/23/2014 at 11:27 PM