SpriteBatch/SpriteFont drawing in 3D

Nov 28, 2013 at 1:25 PM
I am trying to implement text/symbol rendering in 3D perspective projection setup using DirectXTK's SpriteBatch/SpriteFont. I've searched this forum and MSDN on related topics and found two helpful articles [1],[2] given at the end of this post.
To experiment with walbourn's code given in [2], I start with the DirectXTK simple sample [3]. I want to change the code so that the Windows logo and the text "DirectXTK Simple Sample" take on a 3D perspective look. For example, the text is rendered on the surface of the tea pot. The only changes I've made so far is in function Renderer::Render() of file Renderer.cpp. In details, I pass an transform matrix as the last function argument to Renderer::Render. Here is the code:
    // Draw sprite
    XMMATRIX world = XMLoadFloat4x4( &m_world );
    XMMATRIX view = XMLoadFloat4x4( &m_view );
    XMMATRIX projection = XMLoadFloat4x4( &m_projection );
    XMMATRIX local = XMMatrixMultiply( world, XMMatrixTranslation( -2.f, -2.f, -4.f ) );
    XMMATRIX worldViewProj = local * view * projection;

    m_sprites->Begin(DirectX::SpriteSortMode_Deferred, nullptr, nullptr, nullptr, nullptr, nullptr, worldViewProj);
    //m_sprites->Begin();
    m_sprites->Draw( m_texture2.Get(), XMFLOAT2(10,75), nullptr, Colors::White );
    m_font->DrawString( m_sprites.get(), L"DirectXTK Simple Sample", XMFLOAT2( 100, 10 ), Colors::Yellow );
    m_sprites->End();
The rendering result is apparently wrong as both logo and text are invisible at first and then become rays staring from the top-left corner of the screen, radiating rightward and downwards, and then disappear again at the end of animation period. Could someone tell me what's wrong with my code, or how to change the code so that the logo and the text are rendered on the surface of the tea pot or character face? Thank you in advance.

[1] SpriteBatch billboards in a 3D world. http://blogs.msdn.com/b/shawnhar/archive/2011/01/12/spritebatch-billboards-in-a-3d-world.aspx?PageIndex=2.
[2] SpriteBatch.Begin()
https://directxtk.codeplex.com/discussions/459391.
[3] DirectXTK Simple Sample (Windows 8.1)
http://code.msdn.microsoft.com/DirectXTK-Simple-Sample-a0b6de36.
Coordinator
Nov 30, 2013 at 8:09 PM
The best solution for this would be to Render-to-texture with the SpriteFont, and then use that texture to render teapot. This is far easier than trying to use a projection to match the 3D surface...
Nov 30, 2013 at 11:06 PM
walbourn wrote:
The best solution for this would be to Render-to-texture with the SpriteFont, and then use that texture to render teapot. This is far easier than trying to use a projection to match the 3D surface...
Thank you for your reply.
I think I did not present my question correctly when I said "the text is rendered on the surface of the tea pot" in the first thread. I want the text and the tea pot to be rendered independently, that is, the text is still visible even if the tea pot is removed from the scene, or vice visa.
In my experiment with the code sample "DirectXTK Simple Sample", I get the feeling that a SpriteFont object is controlled not only by world-view-projection matrix I provided as the last argument of function m_sprites->Begin(), but also by other transform matrix. For example the third argument of function m_sprites->Draw() control the position of the SpriteFont object, a value of XMFLOAT2(0,0) (XMFLOAT(screenWidth/2, screenHeight/2) resp.) will put the text at the top-left (middle resp.) of the screen. The lack of detailed documentation of DirectX Toolkit API make it more difficult to debug the code.
Nonetheless, I will keep experimenting the code and will look foreword to your further help.
Coordinator
Dec 1, 2013 at 4:15 AM
Edited Dec 1, 2013 at 4:19 AM
SpriteBatch takes positions in screen-space, not world-space.

SpriteFont / SpriteBatch is not really intended as a '3D text' solution. The individual script is not rendered as drawing primitives, but as 'sprites', i.e. 2D textured quads. Getting these to lay out on thte surface of 3D geometry as direct draw operation is not trivial or directly supported by the API.

Assuming you are not targeting Windows phone, you may have better luck with DirectWrite / Direct2D, although that's also not intended as a '3D geometry' font.

It sounds like you want something more akin to a vector font text solution rather than a raster font.
Dec 1, 2013 at 11:51 AM
In article <[email removed]>, "Leonard2012" <[email removed]> writes:
Dec 1, 2013 at 1:34 PM
walbourn wrote:
SpriteBatch takes positions in screen-space, not world-space.

SpriteFont / SpriteBatch is not really intended as a '3D text' solution. The individual script is not rendered as drawing primitives, but as 'sprites', i.e. 2D textured quads. Getting these to lay out on thte surface of 3D geometry as direct draw operation is not trivial or directly supported by the API.

Assuming you are not targeting Windows phone, you may have better luck with DirectWrite / Direct2D, although that's also not intended as a '3D geometry' font.

It sounds like you want something more akin to a vector font text solution rather than a raster font.
Thank you for your help, walbourn.
My proposal to use SpriteFont/SpriteBatch as the 3D text solution is based on the idea that we can transform (rotate/translate) a 2D textured quad so that it is rendered with the desired position and orientation. In fact, this method works in OpenGL Android port of our app. According to your explanation, the primary difficulty is that SpriteFont/SpriteBatch uses screen coordinate, not world coordinate. So could you please develop DirectX Tool Kit to add another version of SpriteFont::Draw that use world space. Add this feature may provide a viable 3D text solution to DirectX11!
Coordinator
Dec 1, 2013 at 11:33 PM
This article uses XNA rather than DirectXTK, but these two SpriteBatch implementations are pretty much identical so same ideas apply to both:

http://blogs.msdn.com/b/shawnhar/archive/2011/01/12/spritebatch-billboards-in-a-3d-world.aspx
Dec 2, 2013 at 10:46 PM
ShawnHargreaves wrote:
This article uses XNA rather than DirectXTK, but these two SpriteBatch implementations are pretty much identical so same ideas apply to both:

http://blogs.msdn.com/b/shawnhar/archive/2011/01/12/spritebatch-billboards-in-a-3d-world.aspx
Thanks, the link is actually the first reference of my original post. Unfortunately, the links in the article is broken. Could you send me a copy of Billboards sample code to my email address at yulq800@hotmail.com so that I can experiment with code. Thanks.
Dec 2, 2013 at 11:31 PM
Hi walbourn, could you explain how did DirectXTK call the shaders. I notice all the shader files are put in the directory DirectXTK Simple Sample\C++\DirectXTK\Src\Shaders, but they are not added to the project, nor they are compiled into .cso files. When I modify a shader file, e.g. SpriteEffect.fx and rebuild the project, nothing changes. There are some files under the directory E:\Projects\DirectX\DirectXTK Simple Sample\C++\DirectXTK\Src\Shaders\Compiled, but I do not know what is the relationship between these files and shader files. As I am experimenting with the Simple Sample code to see if it can be used for 3D text rendering, I need to make some changes to see the effect.
Coordinator
Dec 3, 2013 at 12:49 AM
Edited Dec 3, 2013 at 12:52 AM
DirectXTK includes all it's shader code directly into the static library... The HLSL compiler option /Fh is used to compile the shaders into byte-code that is then written out as a C source file data-array. These are then directly included into the C source code for the DirectXTK library. This avoids any need for including 'stock' CSO files or having to pay for the compilation cost of all the many HLSL shader variants on every rebuild of the C code.

For example, in SpriteBatch.cpp
namespace
{
    // Include the precompiled shader code.
    #include "Shaders/Compiled/SpriteEffect_SpriteVertexShader.inc"
    #include "Shaders/Compiled/SpriteEffect_SpritePixelShader.inc"
The CompileShaders.cmd batch file in DirectXTK\Shaders does this offline compilation.
Marked as answer by walbourn on 1/23/2014 at 11:24 PM
Jan 31, 2014 at 12:18 AM
I’ve come up with a solution for 3D text and sprite rendering based on current implementation of DirectX Tool Kit or SpriteBatch/SpriteFont in particular.
The reason why SpriteBatch/SpriteFont is not capable of drawing 3D text at this moment is that it regards the incoming argument ‘position’ of SpriteBatch::Draw and SpriteFont::DrawString function as coordinate in screen space, not world space, and the implementation code (mainly in SpriteBatch::Impl::RenderSprite) does vertex coordinate transformation in screen space.

Given the above analysis, in order to draw sprite in 3D, we have to do coordinate transformation in world space. Let me explain my implementation.
First of all, I add one more argument ‘sprite3d’ to SpriteBatch::Begin to indicate whether sprites should be drawn in 3D, the declaration is
void XM_CALLCONV Begin(SpriteSortMode sortMode = SpriteSortMode_Deferred, In_opt ID3D11BlendState* blendState = nullptr, In_opt ID3D11SamplerState* samplerState = nullptr, In_opt ID3D11DepthStencilState* depthStencilState = nullptr, In_opt ID3D11RasterizerState* rasterizerState = nullptr, In_opt std::function<void()> setCustomShaders = nullptr, FXMMATRIX transformMatrix = MatrixIdentity, bool sprite3d = false);

Similarly, SpriteBatch::Impl::Begin is changed to
void XM_CALLCONV SpriteBatch::Impl::Begin(SpriteSortMode sortMode, In_opt ID3D11BlendState* blendState, In_opt ID3D11SamplerState* samplerState, In_opt ID3D11DepthStencilState* depthStencilState, In_opt ID3D11RasterizerState* rasterizerState, In_opt std::function<void()> setCustomShaders, FXMMATRIX transformMatrix, bool sprite3d);

Next, in SpriteBatch::Impl::PrepareForRendering(), I remove the viewport transform matrix that is not required in world-view-projection transform.
// Set the viewport transform matrix.
XMMATRIX transformMatrix = mTransformMatrix;
if ( !mSprite3d )
    transformMatrix = mTransformMatrix * GetViewportTransform(deviceContext);
Third, I add a private function RenderSprite3d that differs from RenderSprite in that the vertex data transformation is done in world space.
void SpriteBatch::Impl:: RenderSprite3d(In SpriteInfo const* sprite, _Out_cap_c_(VerticesPerSprite) VertexPositionColorTexture* vertices, FXMVECTOR textureSize, FXMVECTOR inverseTextureSize);

There are still a couple features missing in my implementation, the ‘rotation’, ‘origin’, and ‘effects’ arguments of SpriteBatch::Draw and SpriteFont::DrawString function are not implemented. However, my effort is to show SpriteBatch/SpriteFont can be used as a 3D text solution with some slight modifications. I hope walbourn may add the 3D text feature to DirectX Took Kit in the future so that I do not have to add my patch every time DirectX Took Kit is updated.

Finally, I’ve uploaded the patch to my website at
http://www.eastmodelsoft.com/public/DirectXTKSimpleSamplePatch.zip.

The patch is based on DirectXTK simple sample for Windows 8 (http://code.msdn.microsoft.com/DirectXTK-Simple-Sample-608bc274) and consists of three files. Just replace the corresponding files with files packed in the patch.
Coordinator
Feb 1, 2014 at 5:41 AM
Edited Feb 1, 2014 at 6:37 AM
With the latest version of DirectXTK, there is now a 'SetRotation' mode for SpriteBatch. It would be pretty easy to make SetRotation(DXGI_MODE_ROTATION_UNSPECIFIED) have GetViewportTransform() return identity instead of a viewport. In fact, I just checked a change in to do just that 33999.

As for the rest of the changes, you could do it all with the original transform per ShawnHar's article without really having to write a custom Render function.
Marked as answer by walbourn on 1/31/2014 at 10:41 PM
Feb 1, 2014 at 11:03 PM
I will check the latest release to see if it works as I expected.