使用filesystemwatcher时怎么处理大文件

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了使用filesystemwatcher时怎么处理大文件相关的知识,希望对你有一定的参考价值。

参考技术A 使用filesystemwatcher时怎么处理大文件

FileSystemWatcher基础
属性:
Path——这个属性告诉FileSystemWatcher它需要监控哪条路径。例如,如果我们将这个属性设为“C:\test”,对象就监控test目录下所有文件发生的所有改变(包括删除,修改,创建,重命名)。
IncludeSubDirectories——这个属性说明FileSystemWatcher对象是否应该监控子目录中(所有文件)发生的改变。
Filter——这个属性允许你过滤掉某些类型的文件发生的变化。例如,如果我们只希望在TXT文件被修改/新建/删除时提交通知,可以将这个属性设为“*txt”。在处理高流量或大型目录时,使用这个属性非常方便。
NotifyFilter——获取或设置要监视的更改类型。可以进一步的过滤要监控的更改类型,如watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
事件:
Changed——当被监控的目录中有一个文件被修改时,就提交这个事件。值得注意的是,这个事件可能会被提交多次,即使文件的内容仅仅发生一项改变。这是由于在保存文件时,文件的其它属性也发生了改变。
Created——当被监控的目录新建一个文件时,就提交这个事件。如果你计划用这个事件移动新建的事件,你必须在事件处理器中写入一些错误处理代码,它能处理当前文件被其它进程使用的情况。之所以要这样做,是因为Created事件可能在建立文件的进程释放文件之前就被提交。如果你没有准备正确处理这种情况的代码,就可能出现异常。
Deleted——当被监控的目录中有一个文件被删除,就提交这个事件。
Renamed——当被监控的目录中有一个文件被重命名,就提交这个事件。
注:如果你没有将EnableRaisingEvents设为真,系统不会提交任何一个事件。如果有时FileSystemWatcher对象似乎无法工作,请首先检查EnableRaisingEvents,确保它被设为真。
事件处理
当FileSystemWatcher调用一个事件处理器时,它包含两个自变量——一个叫做“sender”的对象和一个叫做“e”的 FileSystemEventArgs对象。我们感兴趣的自变量为FileSystemEventArgs自变量。这个对象中包含有提交事件的原因。以下是FileSystemEventArgs对象的一些属性:
属性:
Name——这个属性中使事件被提交的文件的名称。其中并不包含文件的路径——只包含使用事件被提交的文件或目录名称。
ChangeType——这是一个WatcherChangeTypes,它指出要提交哪个类型的事件。其有效值包括:
Changed
Created
Deleted
Renamed
FullPath——这个属性中包含使事件被提交的文件的完整路径,包括文件名和目录名。
注意:FileSystemEventArgs对象是监控文件夹下有文件创建、删除、修改时的自变量,如果是重命名的话为RenamedEventArgs对象此时除了FileSystemEventArgs对象的属性值,多了一个OldFullPath,为重命名之前的文件名。

大文件处理
1. Create a new FileSystemWatcher.

1System.IO.FileSystemWatcher fswXmlFileWatcher = new System.IO.FileSystemWatcher(); 2this.fswXmlFileWatcher.EnableRaisingEvents = true; 3this.fswXmlFileWatcher.Path =@"C:\Test" 4//in here I only handle the file created event.5this.fswXmlFileWatcher.Created += newSystem.IO.FileSystemEventHandler(this.fswXmlFileWatcher_Created);

2. Write Code to handle File Created Event.

1private void fswXmlFileWatcher_Created(object sender, System.IO.FileSystemEventArgs e) 2 3lock(this) 4 5string filePath = e.FullPath; 6//incase the file is huge, it need some time to write the whole. so we wait untill the file is accessable by .Net7while (File.GetAttributes(filePath) == FileAttributes.Offline) 8 9Thread.Sleep(500); 10 11//Ok. We start a new thread to process the file12ImportHelper ih = new ImportHelper(filePath); 13Thread th = new Thread(newThreadStart(ih.ImportXmlFile)); 14th.Start(); 15 16

3. what is in the ImportHelper Class.

1public class ImportHelper 2 3private string _filePath; 4private long _fileSize; 5private FileInfo _fileInfo; 6 7public ImportHelper(string filePath) 8 9this._filePath = filePath; 10this._fileSize = 0; 11this._fileInfo = newFileInfo(this._filePath); 12 13 14//we need to this method. same reason. cause the file size is huge. 15private bool TestFile() 16 17long size = this._fileInfo.Length; 18if( this._fileSize == size ) 19 20return true; 21 22else 23 24this._fileSize =size; 25return false; 26 27 28 29public void ImportXmlFile() 30 31while(!this.TestFile() ) 32 33Thread.Sleep(2000); 34 35//It is ok now. The file is ready for us to process. 36//Write your own function to process the file.37ProcessMyFile(this._filePath) 38 39

在一次文本文件变化的时候OnChanged事件会触发两次,这是因为除了文本内容变化之外还有文件其他的属性也变化了例如修改时间。
为了解决这问题,也便于项目当中实际使用,写了下面几个类来实际使用:
主方法:
01 using System;
02 using System.IO;
03
04 namespace test
05
06 class Program
07
08 static void Main(string[] args)
09
10
11
12
13 MyFileSystemWather myWather = new MyFileSystemWather(@"C:\test", "*.txt");
14 myWather.OnChanged += new FileSystemEventHandler(OnChanged);
15 myWather.OnCreated += new FileSystemEventHandler(OnCreated);
16 myWather.OnRenamed += new RenamedEventHandler(OnRenamed);
17 myWather.OnDeleted += new FileSystemEventHandler(OnDeleted);
18 myWather.Start();
19 //由于是控制台程序,加个输入避免主线程执行完毕,看不到监控效果
20 Console.ReadKey();
21
22
23
24 private static void OnCreated(object source, FileSystemEventArgs e)
25
26
27 Console.WriteLine("文件新建事件处理逻辑");
28
29
30
31 private static void OnChanged(object source, FileSystemEventArgs e)
32
33
34 Console.WriteLine("文件改变事件处理逻辑");
35
36
37 private static void OnDeleted(object source, FileSystemEventArgs e)
38
39
40 Console.WriteLine("文件删除事件处理逻辑");
41
42
43 private static void OnRenamed(object source, RenamedEventArgs e)
44
45
46 Console.WriteLine("文件重命名事件处理逻辑");
47
48
49
50
WatcherProcess类:
01 using System.IO;
02
03 namespace test
04
05 public class WatcherProcess
06
07 private object sender;
08 private object eParam;
09
10 public event RenamedEventHandler OnRenamed;
11 public event FileSystemEventHandler OnChanged;
12 public event FileSystemEventHandler OnCreated;
13 public event FileSystemEventHandler OnDeleted;
14 public event Completed OnCompleted;
15
16 public WatcherProcess(object sender, object eParam)
17
18 this.sender = sender;
19 this.eParam = eParam;
20
21
22 public void Process()
23
24 if (eParam.GetType() == typeof(RenamedEventArgs))
25
26 OnRenamed(sender, (RenamedEventArgs)eParam);
27 OnCompleted(((RenamedEventArgs)eParam).FullPath);
28
29 else
30
31 FileSystemEventArgs e = (FileSystemEventArgs)eParam;
32 if (e.ChangeType == WatcherChangeTypes.Created)
33
34 OnCreated(sender, e);
35 OnCompleted(e.FullPath);
36
37 else if (e.ChangeType == WatcherChangeTypes.Changed)
38
39 OnChanged(sender, e);
40 OnCompleted(e.FullPath);
41
42 else if (e.ChangeType == WatcherChangeTypes.Deleted)
43
44 OnDeleted(sender, e);
45 OnCompleted(e.FullPath);
46
47 else
48
49 OnCompleted(e.FullPath);
50
51
52
53
54
MyFileSystemWather类:
001 using System;
002 using System.Collections;
003 using System.IO;
004 using System.Threading;
005
006 namespace test
007
008
009 public delegate void Completed(string key);
010
011 public class MyFileSystemWather
012
013 private FileSystemWatcher fsWather;
014
015 private Hashtable hstbWather;
016
017 public event RenamedEventHandler OnRenamed;
018 public event FileSystemEventHandler OnChanged;
019 public event FileSystemEventHandler OnCreated;
020 public event FileSystemEventHandler OnDeleted;
021
022 ///
023 /// 构造函数
024 ///
025 /// 要监控的路径
026 public MyFileSystemWather(string path, string filter)
027
028 if (!Directory.Exists(path))
029
030 throw new Exception("找不到路径:" + path);
031
032
033 hstbWather = new Hashtable();
034
035 fsWather = new FileSystemWatcher(path);
036 // 是否监控子目录
037 fsWather.IncludeSubdirectories = false;
038 fsWather.Filter = filter;
039 fsWather.Renamed += new RenamedEventHandler(fsWather_Renamed);
040 fsWather.Changed += new FileSystemEventHandler(fsWather_Changed);
041 fsWather.Created += new FileSystemEventHandler(fsWather_Created);
042 fsWather.Deleted += new FileSystemEventHandler(fsWather_Deleted);
043
044
045 ///
046 /// 开始监控
047 ///
048 public void Start()
049
050 fsWather.EnableRaisingEvents = true;
051
052
053 ///
054 /// 停止监控
055 ///
056 public void Stop()
057
058 fsWather.EnableRaisingEvents = false;
059
060
061 ///
062 /// filesystemWatcher 本身的事件通知处理过程
063 ///
064 ///
065 ///
066 private void fsWather_Renamed(object sender, RenamedEventArgs e)
067
068 lock (hstbWather)
069
070 hstbWather.Add(e.FullPath, e);
071
072
073 WatcherProcess watcherProcess = new WatcherProcess(sender, e);
074 watcherProcess.OnCompleted += new Completed(WatcherProcess_OnCompleted);
075 watcherProcess.OnRenamed += new RenamedEventHandler(WatcherProcess_OnRenamed);
076 Thread thread = new Thread(watcherProcess.Process);
077 thread.Start();
078
079
080 private void WatcherProcess_OnRenamed(object sender, RenamedEventArgs e)
081
082 OnRenamed(sender, e);
083
084
085 private void fsWather_Created(object sender, FileSystemEventArgs e)
086
087 lock (hstbWather)
088
089 hstbWather.Add(e.FullPath, e);
090
091 WatcherProcess watcherProcess = new WatcherProcess(sender, e);
092 watcherProcess.OnCompleted += new Completed(WatcherProcess_OnCompleted);
093 watcherProcess.OnCreated += new FileSystemEventHandler(WatcherProcess_OnCreated);
094 Thread threadDeal = new Thread(watcherProcess.Process);
095 threadDeal.Start();
096
097
098 private void WatcherProcess_OnCreated(object sender, FileSystemEventArgs e)
099
100 OnCreated(sender, e);
101
102
103 private void fsWather_Deleted(object sender, FileSystemEventArgs e)
104
105 lock (hstbWather)
106
107 hstbWather.Add(e.FullPath, e);
108
109 WatcherProcess watcherProcess = new WatcherProcess(sender, e);
110 watcherProcess.OnCompleted += new Completed(WatcherProcess_OnCompleted);
111 watcherProcess.OnDeleted += new FileSystemEventHandler(WatcherProcess_OnDeleted);
112 Thread tdDeal = new Thread(watcherProcess.Process);
113 tdDeal.Start();
114
115
116 private void WatcherProcess_OnDeleted(object sender, FileSystemEventArgs e)
117
118 OnDeleted(sender, e);
119
120
121 private void fsWather_Changed(object sender, FileSystemEventArgs e)
122
123 if (e.ChangeType == WatcherChangeTypes.Changed)
124
125 if (hstbWather.ContainsKey(e.FullPath))
126
127 WatcherChangeTypes oldType = ((FileSystemEventArgs)hstbWather[e.FullPath]).ChangeType;
128 if (oldType == WatcherChangeTypes.Created || oldType == WatcherChangeTypes.Changed)
129
130 return;
131
132
133
134
135 lock (hstbWather)
136
137 hstbWather.Add(e.FullPath, e);
138
139 WatcherProcess watcherProcess = new WatcherProcess(sender, e);
140 watcherProcess.OnCompleted += new Completed(WatcherProcess_OnCompleted);
141 watcherProcess.OnChanged += new FileSystemEventHandler(WatcherProcess_OnChanged);
142 Thread thread = new Thread(watcherProcess.Process);
143 thread.Start();
144
145
146 private void WatcherProcess_OnChanged(object sender, FileSystemEventArgs e)
147
148 OnChanged(sender, e);
149
150
151 public void WatcherProcess_OnCompleted(string key)
152
153 lock (hstbWather)
154
155 hstbWather.Remove(key);
156
157
158
159
使用了线程安全的Hashtable来处理一次改变触发两次事件的问题,要注意的是在实际项目使用中,在通过监控文件事情触发时开一个线程WatcherProcess去处理自己业务逻辑的时候,不管业务逻辑成功或者失败(例如有异常抛出一定要try一下)一定要让WatcherProcess的 Completed也就是MyFileSystemWather的WatcherProcess_OnCompleted执行去移除对应变化文件的Hashtable的key,不然下次此文件改变时是无法触发你的业务逻辑的。
还有就是在进行文件监控的时候, 被监控文件在写入的时候,是会有I/O冲突的,即使写入文件是FileShare.Read的也会出现,要真正解决貌似只有FileMaping方法,但是我的项目中文本的写入软件不是我们能控制的,所以只有用处理异常的方法来解决。

使用FileSystemWatcher警告多个文件是否同时更改

我对powershell很新。以下代码由BigTeddy创建,他获得了全部功劳(我还使用While循环进行了一些更改)

我想知道如何创建if / else语句,以便如果同时更改/编辑/创建/删除多个文件(比如同时编辑了10个文件),将创建一个日志文件,说明这些列表文件已在此特定时间同时编辑。

以下powershell脚本BigTeddy创建的基本上是在更改/编辑/创建/删除时,更改时间和编辑了哪些文件时发出日志文件(以及powershell ISE的输出)。

 param(
        [string]$folderToWatch = "C:\Users\gordon\Desktop\powershellStart"
      , [string]$filter        = "*.*"
      , [string]$logFile       = 'C:\Users\gordon\Desktop\powershellDest\filewatcher.log'
    )

    # In the following line, you can change 'IncludeSubdirectories to $true if required.
    $fsw = New-Object IO.FileSystemWatcher $folderToWatch, $filter -Property @{IncludeSubdirectories = $false;NotifyFilter = [IO.NotifyFilters]'FileName, LastWrite'}
    $timeStamp           #My changes
    $timeStampPrev = $timeStamp         #My changes
# This script block is used/called by all 3 events and:
# appends the event to a log file, as well as reporting the event back to the console
$scriptBlock = {

  # REPLACE THIS SECTION WITH YOUR PROCESSING CODE
  $logFile = $event.MessageData # message data is how we pass in an argument to the event script block
  $name = $Event.SourceEventArgs.Name
  $changeType = $Event.SourceEventArgs.ChangeType
  $timeStamp = $Event.TimeGenerated
  while($timeStampPrev -eq $timeStamp) {     #My changes
  Write-Host "$timeStamp|$changeType|'$name'" -fore green
  Out-File -FilePath $logFile -Append -InputObject "$timeStamp|$changeType|'$name'"
  # REPLACE THIS SECTION WITH YOUR PROCESSING CODE
}
}


# Here, all three events are registered.  You need only subscribe to events that you need:
Register-ObjectEvent $fsw Created -SourceIdentifier FileCreated -MessageData $logFile -Action $scriptBlock
Register-ObjectEvent $fsw Deleted -SourceIdentifier FileDeleted -MessageData $logFile -Action $scriptBlock
Register-ObjectEvent $fsw Changed -SourceIdentifier FileChanged -MessageData $logFile -Action $scriptBlock

# To stop the monitoring, run the following commands:
#  Unregister-Event FileDeleted  ;  Unregister-Event FileCreated  ;  Unregister-Event FileChanged


#This script uses the .NET FileSystemWatcher class to monitor file events in folder(s).
#The advantage of this method over using WMI eventing is that this can monitor sub-folders.
#The -Action parameter can contain any valid Powershell commands.
#The script can be set to a wildcard filter, and IncludeSubdirectories can be changed to $true.
#You need not subscribe to all three types of event.  All three are shown for example.
答案

您是否考虑过将时间戳和操作(创建/修改/删除)作为键和文件名作为值的哈希表。您在特定等待间隔后迭代字典,并将字典中的条目刷新到日志文件。

以上是关于使用filesystemwatcher时怎么处理大文件的主要内容,如果未能解决你的问题,请参考以下文章

c# filesystemwatcher 监控文件是不是写完

C# FileSystemWatcher - 多个事件

c# FileSystemWatcher如何处理多个文件?

FileSystemWatcher - 在删除时,复制一个文件

FileSystemWatcher .Net 3.5

关于FileSystemWatcher监听文件创建