[jcifs] Peformance questions

[Philip Warner]

Hi,

For brevity, the summary is I have two questions regarding saving a file to a windows (Samba) share:
*
Q1: C**an I reduce the number of times it calls peekKey()?*
*Q2: Is there any optimization that I can perform to reduce the DfsResolve() calls or cost?*


The background for these questions is below...

I am quite new to jCIFS. I recently added it to my Android app (Book Catalogue) and was surprised at how long it took to write data
to a network share via a WiFi link. I did some searches and found various references to setting buffer sizes as well as copying
data, but these did not seem to help (or explain the slowness completely).

For some numbers:

- save locally (sdcard) approx 1 minute (80mb file)
- save to network approx 5min 30sec (80mb file)
- copy file from local to network (using ES File Explorer), approx 1 min 30 (80mb file)

My next step was to profile the app. In a 29 second profiling session (much longer risks crashing either Android or the app), I found:

- 28,972ms total time
- 4007 ms total CPU (so it's spending 85% of its time waiting)
- of the total time, 53.7% (15571ms) is spent in SmbTransport.peekKey(). It was called a total of 337 times in the 29 seconds.

I assume that the peekKey() time is waiting for an ACK of some kind.

So the first question is:

*Q1: C**an I reduce the number of times it calls peekKey()?*

Once I ignore the peekKey() related calls (Transport.readn(), PlainSocketInputStream.read() etc), the next biggest time is:

- 7371ms in SmbFileOutputStream.writeDirect()

This does not worry me overly; it spends approx 26% of the total time writing the file; if it was reduced to this only, then the
total time would go from 5.5 min to less than 1.5 min, which is what I see from 'ES File Explorer' copying a file.

The next one I see that is quite odd is:

- 5734 ms in Dfs.resolve() called from SmbFile.send(), called 168 times in 29 seconds.

It's no nig deal, but it does account for 78% of the time in SmbFile.send(), and does seem to actually ask the OS for name
translation on each call (there are 168 calls to InetAddress.getAllByNameImpl()).

Which leads to my second question:

*Q2: Is there any optimization that I can perform to reduce the DfsResolve() calls or cost?*


I may of course be barking up the wrong tree here, this may be a simple unavoidable result of using SMB, but the initial stats (file
copy time using a different app) leads me to believe otherwise.

Any help would be greatly appreciated!

Thanks.

[Michael B Allen]

Hi Philip,

It sounds like there's some kind of networking quirk like a socket
option that's wrong for the Android JVM. This almost certainly has
little or nothing to do with JCIFS. It's not a "performance" issue.
The methods that you reference are just sleeping. Although there could
easily be something timing out like a DFS referral that isn't
applicable because you're not in a domain environment in which case
you should disable DFS with jcifs.smb.client.dfs.disabled = false.
Otherwise, I'm not familiar with Android development so I have no idea
as to how to go about debugging such a thing.

Mike

Post by Philip Warner
Hi,
For brevity, the summary is I have two questions regarding saving a file to
Q1: Can I reduce the number of times it calls peekKey()?
Q2: Is there any optimization that I can perform to reduce the DfsResolve()
calls or cost?
The background for these questions is below...
I am quite new to jCIFS. I recently added it to my Android app (Book
Catalogue) and was surprised at how long it took to write data to a network
share via a WiFi link. I did some searches and found various references to
setting buffer sizes as well as copying data, but these did not seem to help
(or explain the slowness completely).
- save locally (sdcard) approx 1 minute (80mb file)
- save to network approx 5min 30sec (80mb file)
- copy file from local to network (using ES File Explorer), approx 1 min 30 (80mb file)
My next step was to profile the app. In a 29 second profiling session (much
- 28,972ms total time
- 4007 ms total CPU (so it's spending 85% of its time waiting)
- of the total time, 53.7% (15571ms) is spent in SmbTransport.peekKey(). It
was called a total of 337 times in the 29 seconds.
I assume that the peekKey() time is waiting for an ACK of some kind.
Q1: Can I reduce the number of times it calls peekKey()?
Once I ignore the peekKey() related calls (Transport.readn(),
- 7371ms in SmbFileOutputStream.writeDirect()
This does not worry me overly; it spends approx 26% of the total time
writing the file; if it was reduced to this only, then the total time would
go from 5.5 min to less than 1.5 min, which is what I see from 'ES File
Explorer' copying a file.
- 5734 ms in Dfs.resolve() called from SmbFile.send(), called 168 times in 29 seconds.
It's no nig deal, but it does account for 78% of the time in SmbFile.send(),
and does seem to actually ask the OS for name translation on each call
(there are 168 calls to InetAddress.getAllByNameImpl()).
Q2: Is there any optimization that I can perform to reduce the DfsResolve()
calls or cost?
I may of course be barking up the wrong tree here, this may be a simple
unavoidable result of using SMB, but the initial stats (file copy time using
a different app) leads me to believe otherwise.
Any help would be greatly appreciated!
Thanks.

[Philip Warner]

Hi,

Thanks for the prompt reply. While in some sense sleeping is not a performance issue per-se, if it spends 85% of it's time sleeping
while the app waits, it's at least a perceived performance issue. But that's just nit-picking.

Post by Michael B Allen
you should disable DFS with jcifs.smb.client.dfs.disabled

had a remarkable effect. The time waiting for peekKey() went down to 20% of total time, and total time dropped by 75% from 5min30 to
1min17m. The total time in SmbWrite dropped from 7secs to 2sec.

So...this leads to two more questions:

Q3: is there any disadvantage in disabling DFS permanently?

Q4: in general terms, if this were a Windows or Linux setup, how would you go about working out what was wrong with having DFS
enabled? FWIW, I can run a packet sniffer on a Linux server if that helps.


Thanks again!

Post by Michael B Allen
It sounds like there's some kind of networking quirk like a socket
option that's wrong for the Android JVM. This almost certainly has
little or nothing to do with JCIFS. It's not a "performance" issue.
The methods that you reference are just sleeping. Although there could
easily be something timing out like a DFS referral that isn't
applicable because you're not in a domain environment in which case
you should disable DFS with jcifs.smb.client.dfs.disabled = false.
Otherwise, I'm not familiar with Android development so I have no idea
as to how to go about debugging such a thing.
Mike

[Michael B Allen]

Post by Philip Warner
Hi,
Thanks for the prompt reply. While in some sense sleeping is not a

performance issue per-se, if it spends 85% of it's time sleeping

Post by Philip Warner
while the app waits, it's at least a perceived performance issue. But

that's just nit-picking.

Post by Philip Warner

Post by Michael B Allen
you should disable DFS with jcifs.smb.client.dfs.disabled

had a remarkable effect. The time waiting for peekKey() went down to 20%

of total time

Hi Philip,

peekKey is called while waiting for a response to a request. So if you
eliminate some request that is just timing out then naturally the
"performance" is going to look a lot better.

Post by Philip Warner
, and total time dropped by 75% from 5min30 to
1min17m. The total time in SmbWrite dropped from 7secs to 2sec.
Q3: is there any disadvantage in disabling DFS permanently?

Well if you try to access a dfs volume then it's not going to work. But if
your using it in a non-corporate home pc type environment then you could
disable dfs no problem.

Post by Philip Warner
Q4: in general terms, if this were a Windows or Linux setup, how would

you go about working out what was wrong with having DFS

Post by Philip Warner
enabled? FWIW, I can run a packet sniffer on a Linux server if that helps.

You really need a capture from android. And a capture is what you really
need to figure out the root of the problem.

To be honest now it sounds like jcifs is doing something wrong. It probably
shouldn't hang if the server or env doesn't support dfs. And the only way
to really figure out what the problem is is to get a capture.

Note: do not post captures to the list. If you send me a capture directly
I'll look at it (eventually).

Mike

Post by Philip Warner
Thanks again!

Post by Michael B Allen
It sounds like there's some kind of networking quirk like a socket
option that's wrong for the Android JVM. This almost certainly has
little or nothing to do with JCIFS. It's not a "performance" issue.
The methods that you reference are just sleeping. Although there could
easily be something timing out like a DFS referral that isn't
applicable because you're not in a domain environment in which case
you should disable DFS with jcifs.smb.client.dfs.disabled = false.
Otherwise, I'm not familiar with Android development so I have no idea
as to how to go about debugging such a thing.
Mike

[Philip Warner]

You really need a capture from android. And a capture is what you really need to figure out the root of the problem.
To be honest now it sounds like jcifs is doing something wrong. It probably shouldn't hang if the server or env doesn't support
dfs. And the only way to really figure out what the problem is is to get a capture.

Hi Mike,

Thanks again; it looks like tcpdump may be available on Android. Would that be sufficient? If so, can you suggest the relevant
command to get what you need?

[Philip Warner]

To be honest now it sounds like jcifs is doing something wrong. It probably shouldn't hang if the server or env doesn't support
dfs. And the only way to really figure out what the problem is is to get a capture.

I'd suggest that one thing at least is weird: why does it make DFS enquiries while in the process of writing to a file? ISTM that
once I have an OutputStream it really should not be doing (or needing) DNS stuff...but I don't know CIFS.

[Michael B Allen]

Post by Philip Warner

Post by Michael B Allen
To be honest now it sounds like jcifs is doing something wrong. It

probably shouldn't hang if the server or env doesn't support

Post by Philip Warner

Post by Michael B Allen
dfs. And the only way to really figure out what the problem is is to get a capture.

I'd suggest that one thing at least is weird: why does it make DFS

enquiries while in the process of writing to a file? ISTM that

Post by Philip Warner
once I have an OutputStream it really should not be doing (or needing)

DNS stuff...but I don't know CIFS.
JCIFS should absolutely not be querying DFS during writes and I would be
surprised if it actually is. That is something that would require a proper
capture to verify.

Mike

[Philip Warner]

JCIFS should absolutely not be querying DFS during writes and I would be surprised if it actually is. That is something that would
require a proper capture to verify.

You are probably right about this, but based on 1.3.14 sources:

SmbFileOutputStream.writeDirect(...) calls SmbFile.send(...)
SmbFile.send(...) calls resolveDfs(...)
resolveDfs(...) calls Dfs.resolve(...)

Dfs.resolve(...) looks like it does OS-level network stuff unless DFS is disabled, or unless it has succeeded in a call to
Dfs.resolve(...) recently.

My suspicion is that because DFS is enabled, and because DFS is not present, it is retrying for every block it sends. Hence the 400%
slowdown when DFS is enabled.

Or it could just the the OS-level calls are really slow.

[Michael B Allen]

Post by Philip Warner

Post by Michael B Allen
JCIFS should absolutely not be querying DFS during writes and I would

be surprised if it actually is. That is something that would

Post by Philip Warner

Post by Michael B Allen
require a proper capture to verify.

SmbFileOutputStream.writeDirect(...) calls SmbFile.send(...)
SmbFile.send(...) calls resolveDfs(...)
resolveDfs(...) calls Dfs.resolve(...)

Hi Philip,

The those DFS methods should just return immediately after being called
once. They cache results.

Post by Philip Warner
Dfs.resolve(...) looks like it does OS-level network stuff unless DFS is

disabled, or unless it has succeeded in a call to

Post by Philip Warner
Dfs.resolve(...) recently.
My suspicion is that because DFS is enabled, and because DFS is not

present, it is retrying for every block it sends. Hence the 400%

Post by Philip Warner
slowdown when DFS is enabled.

I doubt very much that is happening. But if you can produce a capture that
shows a problem I'll fix it (eventually).

Post by Philip Warner
Or it could just the the OS-level calls are really slow.

Jcifs does not call any OS specific functions. It is 100% java.

Mike

[Philip Warner]

Post by Philip Warner
My suspicion is that because DFS is enabled, and because DFS is not present, it is retrying for every block it sends. Hence the 400%
slowdown when DFS is enabled.

I doubt very much that is happening. But if you can produce a capture that shows a problem I'll fix it (eventually).

Post by Philip Warner
Or it could just the the OS-level calls are really slow.

Jcifs does not call any OS specific functions. It is 100% java.

I did ask before if you thought tcpdump would be up to the task, and if so, if you had any suggested command etc. It's been a long
time since I played with it.

That said, I just did a quick profile again with DFS enabled. And in the time I ran the profile it did:

- 87 calls to SmbFileOutputStream.write()
- 87 calls to Dfs.resolve()
- 87 calls to Dfs.getTrustedDomains()
- 87 calls to UniAddress.getByName()
- 87 calls to UniAddress.getAllByName()
- 87 calls to java/net/InetAddress.getAllByName()
- 87 calls to java/net/InetAddress.lookupHostByName()
- 87 calls to libcore/io/ForwardingOs.getaddrinfo

which at least demonstrates that caching is not happening and that OS-level calls are happening for every write (I did not mean
OS-specific calls, I did say OS-level calls in the original, and there is a subtle difference).

The reason I think caching is not happening is because it does all the work to setup DFS and fails on every write() call so caches a
'null' value.

I agree I can not prove that network access is is happening without a protocol dump, but I think this is pretty clear evidence that
basic level DFS-related code is being called for every single write().

The fact it goes from 85% idle to 20% idle when DFS is disabled at least suggests that some extra (probably network) IO is happening.

[Michael B Allen]

Post by Philip Warner

Post by Philip Warner
My suspicion is that because DFS is enabled, and because DFS is not present, it is retrying for every block it sends. Hence the 400%
slowdown when DFS is enabled.

I doubt very much that is happening. But if you can produce a capture that shows a problem I'll fix it (eventually).

Post by Philip Warner
Or it could just the the OS-level calls are really slow.

Jcifs does not call any OS specific functions. It is 100% java.

I did ask before if you thought tcpdump would be up to the task, and if so, if you had any suggested command etc. It's been a long
time since I played with it.
- 87 calls to SmbFileOutputStream.write()
- 87 calls to Dfs.resolve()
- 87 calls to Dfs.getTrustedDomains()
- 87 calls to UniAddress.getByName()
- 87 calls to UniAddress.getAllByName()
- 87 calls to java/net/InetAddress.getAllByName()
- 87 calls to java/net/InetAddress.lookupHostByName()
- 87 calls to libcore/io/ForwardingOs.getaddrinfo
which at least demonstrates that caching is not happening and that OS-level calls are happening for every write (I did not mean
OS-specific calls, I did say OS-level calls in the original, and there is a subtle difference).
The reason I think caching is not happening is because it does all the work to setup DFS and fails on every write() call so caches a
'null' value.
I agree I can not prove that network access is is happening without a protocol dump, but I think this is pretty clear evidence that
basic level DFS-related code is being called for every single write().
The fact it goes from 85% idle to 20% idle when DFS is disabled at least suggests that some extra (probably network) IO is happening.

Yes, that is obvious. Like I said before, the socket connect is
probably timing out. The client probably doesn't have access to a DC
for some reason. There might be a logical change that I could make to
dodge that scenario. The client should not stall if it can't
communicate with a DC. But fixing that is not going to happen anytime
soon. It's not an easy change.

And in your case, without any way to lookup DFS roots, DFS support
will be useless anyway. So you might as well just disable DFS and move
on.

Mike

[levin]

jcifs.Config.setProperty("jcifs.smb.client.dfs.disabled", "true");
jcifs.Config.setProperty("jcifs.smb.client.useExtendedSecurity",
"false");
jcifs.Config.setProperty("jjcifs.smb.lmCompatibility", "2");
jcifs.Config.setProperty("jcifs.smb.client.useExtendedSecurity",
"false");
jcifs.Config.setProperty("jcifs.smb.client.soTimeout", "35000");
jcifs.Config.setProperty("jcifs.resolveOrder", "LMHOSTS,BCAST,DNS");



I build a http server in my android device,but it's a little slow.



--
View this message in context: http://samba.2283325.n4.nabble.com/Peformance-questions-tp4644647p4686860.html
Sent from the Samba - jcifs mailing list archive at Nabble.com.

[levin]

i use jcifs 1.3.18



------------------ Original ------------------
From: "levin [via Samba]"<ml-node+***@n4.nabble.com>;
Date: 2015幎6月7日(星期倩) 晚䞊9:26
To: "Levin ++"<***@foxmail.com>;
Subject: Re: Peformance questions



jcifs.Config.setProperty("jcifs.smb.client.dfs.disabled", "true");
jcifs.Config.setProperty("jcifs.smb.client.useExtendedSecurity", "false");
jcifs.Config.setProperty("jjcifs.smb.lmCompatibility", "2");
jcifs.Config.setProperty("jcifs.smb.client.useExtendedSecurity", "false");
jcifs.Config.setProperty("jcifs.smb.client.soTimeout", "35000");
jcifs.Config.setProperty("jcifs.resolveOrder", "LMHOSTS,BCAST,DNS");



I build a http server in my android device,but it's a little slow.


If you reply to this email, your message will be added to the discussion below:
http://samba.2283325.n4.nabble.com/Peformance-questions-tp4644647p4686860.html
To start a new topic under Samba - jcifs, email ml-node+***@n4.nabble.com
To unsubscribe from Peformance questions, click here.
NAML



--
View this message in context: http://samba.2283325.n4.nabble.com/Peformance-questions-tp4644647p4686861.html
Sent from the Samba - jcifs mailing list archive at Nabble.com.