This is the last part of a five part series where I introduce a flexible REST service that works with non-active directory sources. The first part is available here: [http://community.dynamics.com/ax/b/dynamicsax_wpfandnetinnovations/archive/2014/03/08/creating-flexible-restful-services-for-dynamics.aspx].
The previous article, in the series, introduced client-side examples of how to consume the Generic Broker RESTful service. The examples given were simply to illustrate the flexibility of the architecture; however, in reality, all communication should be channelled through a more secure framework. This article will focus on server-side consumption of Dynamics data, specifically from SQL Servers outside of the trusted domain or SQL Servers within the trust, but where linked-server functionality is disabled. A video walkthrough of this server-side implementation can be found here: [https://www.youtube.com/watch?v=h6PqeRqJTZM&hd=1]
· Ordinarily, all client-side access to the Dynamics SQL server is blocked either by locking down access ports or disabling client remote connections. This protects company data from rogue client-side requests from ODBC or MS Query capable applications. Server-to-server communications is better regulated through the use of firewall rules. Data access is from vetted, verified and trusted sources and can be implemented through robust development processes.
Dpending on your requirements, HTTP web-requests can be invoked from SQL Server using either one of two methods: OLE automation or CLR. Yes I know, we’ve done a good job so far *not* writing any CLR, but now we are at the stage where we need to do some real programmingJ.
The standard OLE automation method is shown below. It’s simple to implement, but has an inherent restriction on the size of data that can be received, so therefore is only useful for the retrieval of single record data.
TSQL Scalar function: GetHttp |
sp_configure'show advanced options', 1; GO RECONFIGURE; GO sp_configure'Ole Automation Procedures', 1; GO RECONFIGURE; GO CREATEFUNCTIONGetHttp ( @urlvarchar(8000) ) RETURNSvarchar(8000) AS BEGIN DECLARE@winint DECLARE@hr int DECLARE@textvarchar(8000) EXEC@hr=sp_OACreate'WinHttp.WinHttpRequest.5.1',@winOUT IF@hr<> 0 EXECsp_OAGetErrorInfo@win EXEC@hr=sp_OAMethod@win,'Open',NULL,'GET',@url,'false' IF@hr<> 0 EXECsp_OAGetErrorInfo@win EXEC@hr=sp_OAMethod@win,'Send' IF@hr<> 0 EXECsp_OAGetErrorInfo@win EXEC@hr=sp_OAGetProperty@win,'ResponseText',@textOUTPUT IF@hr<> 0 EXECsp_OAGetErrorInfo@win EXEC@hr=sp_OADestroy@win IF@hr<> 0 EXECsp_OAGetErrorInfo@win SET@text=REPLACE(@text,'<','<'); SET@text=REPLACE(@text,'>','>'); RETURN@text END |
If you need to return large DataSets, then you’ll need to implement a CLR assembly that will allow you to both make web-requests and return lengthy XML data. The remainder of this article will cover this more complex scenario.
Start Visual Studio and create a C# Class-library project and paste in the following code (rename the class according to your naming- convention):
Class: LNPS_SqlServerCLRClass |
using System; using System.Data; using System.Data.SqlClient; using System.Data.SqlTypes; using Microsoft.SqlServer.Server; using System.Net; using System.IO; publicpartialclassLNPS_SqlServerCLRClass { [Microsoft.SqlServer.Server.SqlProcedure] publicstaticvoid HTTPGet(SqlString weburl, [SqlFacet(MaxSize = -1)] outSqlString returnval) { string url = Convert.ToString(weburl); string feedData = string.Empty; try { HttpWebRequest request = null; HttpWebResponse response = null; Stream stream = null; StreamReader streamReader = null; request = (HttpWebRequest)WebRequest.Create(url); request.Method = "GET"; request.ContentLength = 0; response = (HttpWebResponse)request.GetResponse(); stream = response.GetResponseStream(); streamReader = newStreamReader(stream); feedData = streamReader.ReadToEnd(); response.Close(); stream.Dispose(); streamReader.Dispose(); } catch (Exception ex) { SqlContext.Pipe.Send(ex.Message.ToString()); } feedData = feedData.Replace("<", "<"); feedData = feedData.Replace(">", ">"); returnval = feedData; } } |
The above C# code initiates a standard HttpWebRequest and returns the data as a SqlString object. Note the use of “[SqlFacet(MaxSize = -1)]” in the parameter return type. This allows a mapping from the .net “String” data type to the SQL “nvarchar(max)” data type. This sets the theoretical upper limit to 2GB for the returned XML string. This should be ample for most data harvesting requirements. Any more than that and you should be thinking of using offline synchronisation processes (like the Dynamics Connector).
Compile the project and copy the new assembly to a suitable [Bin] folder on the target SQL server. I’ve chosen the [Tools] to keep it separate from the standard assemblie (e.g. \\server\c$\Program Files\Microsoft SQL Server\100\Tools\Binn).
Select the database, from where you will be writing your harvesting scripts. You will now need to issue a series of administration directives on this database before you can start using the new assembly. Not all of these will be required depending on your security configuration:
Give yourself the authority to make the necessary changes on the database. |
ALTERAUTHORIZATIONONDatabase::[DATABASE] TO [DOMAIN\USER] |
Note: Alternatively, login to SQL as the database administrator. |
Enable your database for CLR. |
EXECsp_configure'clr enabled', 1 GO RECONFIGURE GO EXECsp_configure'clr enabled' GO |
Allow the SQL instance to trust the database and its contents. |
ALTERDATABASE [DATABASE] SETTRUSTWORTHYON |
Note: Check with your SQL administrator to confirm that this doesn’t contravene policy. |
Install the assembly into the current database. |
CREATEASSEMBLY LNPS_SqlServerCLR FROM'D:\Program Files\Microsoft SQL Server\100\Tools\Binn\LNPS_SqlServerCLR.dll' WITHPERMISSION_SET=UNSAFE; |
Note: Choose the path where you installed your custom assembly. |
Create a stored procedure wrapper to invoke the CLR method |
CREATEPROCEDURE [dbo].[HTTPGet] @WebUrl [nvarchar](4000), @ReturnVal [nvarchar](MAX)OUTPUT WITHEXECUTEASCALLER AS EXTERNAL NAME [LNPS_SqlServerCLR].[LNPS_SqlServerCLRClass].[HTTPGet] |
Note: Fully qualified path is [namespace].[class].[method] |
Test the new CLR by performing a HTTP GET against any website |
DECLARE @response NVARCHAR(MAX) EXECUTE HTTPGet'http://www.microsoft.com',@Response OUT SELECT @response |
Note: This will return the HTML from the chosen website. Your SQL server will need internet access for this to work. |
We are now ready to make web-requests from our database!
Switch back to Dynamics and create a job that contains the following X++. The “While-Select” statement returns some basic information from the [CustTable]:
X++ |
CustTable custTable; whileselect * from custTable where custTable.CustGroup == 'EU' { info(strFmt("AccountNum = %1, Name = %2, Balance = %3", custTable.AccountNum, custTable.name(), custTable.balanceAllCurrency('GBP'))); } |
You now need to convert this to a simple (attribute-based) XML structure, for consumption by SQL server. This is done by reformatting the output:
CustTable custTable; str result = '<Root>'; whileselect * from custTable where custTable.CustGroup == 'EU' { result += '<CustTable AccountNum="' + custTable.AccountNum + '" Name="' + custTable.name() + '" Balance="' + num2str(custTable.balanceAllCurrency('GBP'), 0, 2, 1, 0) + '" />'; } result += '</Root>'; return result; |
Sample results of the above script are shown below: |
<?xmlversion="1.0"encoding="utf-8"?> <Root> <CustTableAccountNum="00000066"Name="EU Customer 1"Balance="2866.50" /> <CustTableAccountNum="00000070"Name="EU Customer 2"Balance="10710.00" /> <CustTableAccountNum="00000085"Name="EU Customer 3"Balance="684.00" /> <CustTableAccountNum="00000087"Name="EU Customer 4"Balance="2374.00" /> </Root> |
This XML data packet is now compatible with the OPENXML command in TSQL.
· The OPENXML command provides a rowset view over an XML document, allowing it to be used in TSQL statements. More information on the OPENXML command can be found here: [http://technet.microsoft.com/en-us/library/ms186918.aspx]
The last stage is split into three parts:
1.) Transfer the X++ to SQL.
2.) Invoke the passthru X++ using our REST service.
3.) Convert the XML response into a SQL DataSet.
Transfer the X++ to SQL. |
DECLARE @idoc ASINT, @script ASNVARCHAR(2000), @request ASNVARCHAR(4000), @response ASNVARCHAR(MAX); SET @script =N'<![CDATA[ CustTable custTable; str result = ''<Root>''; while select * from custTable where custTable.CustGroup == ''DOM'' { result += ''<CustTable AccountNum="'' + custTable.AccountNum + ''" Name="'' + custTable.name() + ''" Balance="'' + num2str(custTable.balanceAllCurrency(''GBP''), 0, 2, 1, 0) + ''" />''; } result += ''</Root>''; return result; ]]>'; SET @script =REPLACE(@script,'+','%2B'); |
Invoking the passthru X++ using the REST service. |
SET @request ='http://vmssp-01/GenericBroker/Service.asmx/ProcessImmediate?requestXml=<brokerRequest><direction>SQL->L1</direction><serviceClass>DynamicsClass</serviceClass><invocationMethod>ExecuteScript</invocationMethod><data>'+ @script +'</data><resultOnly>true</resultOnly></brokerRequest>'; EXECUTE HTTPGet@request, @response OUT |
Convert the XML result into a SQL DataSet. |
-- Create an internal representation of the XML document. EXECsp_xml_preparedocument@idoc OUTPUT, @response; -- Execute a SELECT statement that uses the OPENXML rowset provider. SELECT * FROM OPENXML (@idoc,'/Root/CustTable',1) WITH (AccountNum nvarchar(20), Name varchar(50), Balance decimal(13,2)); |
Note: Choose the Xpath and components of the OpenXML dataset must match the attribute tags within the XML. |
Voila, you have just executed X++ inside your TSQL!
The output is fundamentally a SQL DataSet and can be used and manipulated as you would with any other table within SQL.You can modify any part of the X++ and execute the script immediately without the need to recompile any of the objects within the end-to-end solution…
This means that a 3rd party can implement integrations into Dynamics without having to create and deploy point-specific WCF assemblies. Additionally, the integration can be extended without the need to modify AOT queries and recreate associated AIF document services. Overall, the change impact on Dynamics decreases and the flexibility at the other end of the pipe increases.
Have fun with this and let me know how you guys get on!
REGARDS