我有大小约为 212 MB 的 JSON blob 文件。
在本地调试时,下载大约需要 15 分钟。
当我将代码部署到 Azure 应用程序服务时,它运行了 10 分钟并失败并出现错误:(在本地,它间歇性失败并出现相同的错误)
服务器无法验证请求。确保值
授权标头格式正确,包括签名
代码尝试1:
// Create SAS Token for referencing a file for a duration of 5 min
SharedAccessBlobPolicy sasConstraints = new SharedAccessBlobPolicy
{
SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(15),
Permissions = SharedAccessBlobPermissions.Read
};
var blob = cloudBlobContainer.GetBlockBlobReference(blobFilePath);
string sasContainerToken = blob.GetSharedAccessSignature(sasConstraints);
var cloudBlockBlob = new CloudBlockBlob(new Uri(blob.Uri + sasContainerToken));
using (var stream = new MemoryStream())
{
await cloudBlockBlob.DownloadToStreamAsync(stream);
//resetting stream's position to 0
stream.Position = 0;
var serializer = new JsonSerializer();
using (var sr = new StreamReader(stream))
{
using (var jsonTextReader = new JsonTextReader(sr))
{
jsonTextReader.SupportMultipleContent = true;
result = new List<T>();
while (jsonTextReader.Read())
{
result.Add(serializer.Deserialize<T>(jsonTextReader));
}
}
}
}
代码尝试 2:我尝试使用 DownloadRangeToStreamAsync 来下载块中的 blob,但没有任何改变:
int bufferLength = 1 * 1024 * 1024;//1 MB chunk
long blobRemainingLength = blob.Properties.Length;
Queue<KeyValuePair<long, long>> queues = new Queue<KeyValuePair<long, long>>();
long offset = 0;
do
{
long chunkLength = (long)Math.Min(bufferLength, blobRemainingLength);
offset += chunkLength;
blobRemainingLength -= chunkLength;
using (var ms = new MemoryStream())
{
await blob.DownloadRangeToStreamAsync(ms, offset, chunkLength);
ms.Position = 0;
lock (outPutStream)
{
outPutStream.Position = offset;
var bytes = ms.ToArray();
outPutStream.Write(bytes, 0, bytes.Length);
}
}
}
while (blobRemainingLength > 0);
我认为 212 MB 数据并不是一个大的 JSON 文件。你能推荐一个
解决方案 ?
我建议你可以尝试使用Azure 存储数据移动库 https://github.com/Azure/azure-storage-net-data-movement.
我用220MB大小的较大文件进行了测试,将其下载到内存大约需要5分钟。
示例代码:
SharedAccessBlobPolicy sasConstraints = new SharedAccessBlobPolicy
{
SharedAccessExpiryTime = DateTime.UtcNow.AddMinutes(15),
Permissions = SharedAccessBlobPermissions.Read
};
CloudBlockBlob blob = blobContainer.GetBlockBlobReference("t100.txt");
string sasContainerToken = blob.GetSharedAccessSignature(sasConstraints);
var cloudBlockBlob = new CloudBlockBlob(new Uri(blob.Uri + sasContainerToken));
var stream = new MemoryStream();
//set this value as per your need
TransferManager.Configurations.ParallelOperations = 5;
Console.WriteLine("begin to download...");
//use Stopwatch to calculate the time
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
DownloadOptions options = new DownloadOptions();
options.DisableContentMD5Validation = true;
//use these lines of code just for checking the downloading progress, you can remove it in your code.
SingleTransferContext context = new SingleTransferContext();
context.ProgressHandler = new Progress<TransferStatus>((progress) =>
{
Console.WriteLine("Bytes downloaded: {0}", progress.BytesTransferred);
});
var task = TransferManager.DownloadAsync(cloudBlockBlob, stream,options,context);
task.Wait();
stopwatch.Stop();
Console.WriteLine("the length of the stream is: "+stream.Length);
Console.WriteLine("the time is taken: "+stopwatch.ElapsedMilliseconds);
测试结果:
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)