Comments on adding custom .Net Activity for Azure Data Factory

Hello,

The tutorial Microsoft provides for getting started with Azure Data Factory is great!

I was interested in adding custom .Net activity to Azure Data Factory Pipeline and I checked this tutorial .

This tutorial is also very good, but I needed to make small modifications.

I. SQL Issues

The first issue was related with missing clustered index.

The error I got when the pipeline was running was:

“Database operation failed on server ‘{database-server}.database.windows.net,1433’ with SQL Error Number ‘40054’. Error message from database execution : Tables without a clustered index are not supported in this version of SQL Server. Please create a clustered index and try again..”

And the fix was:


CREATE UNIQUE CLUSTERED INDEX Idx_emp ON emp(ID);

II. Azure Storage Issues

The second issue was related to the version of the Azure Storage .Net SDK library. I used the latest which was version 5.0.2

However,I got the following error:

Unknown error in module: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.TypeLoadException: Could not load type ‘Microsoft.WindowsAzure.Storage.Blob.CloudBlob’ from assembly ‘Microsoft.WindowsAzure.Storage, Version=4.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’. at MyDotNetActivityNS.MyDotNetActivity.Execute(IEnumerable`1 linkedServices, IEnumerable`1 tables, Activity activity, IActivityLogger logger) at Microsoft.Azure.Management.DataFactories.Runtime.ActivityExecutor.Execute(Object job, String configuration, Action`1 logAction) — End of inner exception stack trace — at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) at Microsoft.DataPipeline.Compute.HDInsightJobExecution.ReflectingActivityWrapper.Execute() in d:\_Bld\8809\6742\Sources\Product\Common\Compute\src\HDIComputeDelegatorJob\ReflectingActivityWrapper.cs:line 44 at Microsoft.DataPipeline.Compute.HDInsightJobExecution.JobWrapper.RunJob() in d:\_Bld\8809\6742\Sources\Product\Common\Compute\src\HDIComputeDelegatorJob\JobWrapper.cs:line 94 at Microsoft.DataPipeline.Compute.HDInsightJobExecution.Launcher.Main(String[] args) in d:\_Bld\8809\6742\Sources\Product\Common\Compute\src\HDIComputeDelegatorJob\Launcher.cs:line 78.

I spotted that in the message was mentioned ‘Microsoft.WindowsAzure.Storage, Version=4.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ so I decided to downgrade to this version with the following command in the Nuget Packet Manager Console in Visual Studio:

PM> Install-Package WindowsAzure.Storage -Version 4.3.0

It looks like Azure Data Factory is currently dependent on version 4.3.0 of Azure Storage Nuget of the Azure Storage SDK and it is not working with the latest version.

I hope these two small fixes will save time for other people as well.

Thank you!

Kanio

Advertisements

2 thoughts on “Comments on adding custom .Net Activity for Azure Data Factory

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s