So while I was writing that tutorial posted here, I was playing around in Starcraft. I noticed a couple functions that had funny calling conventions. For example, I notice parameters being passed in eax, occasionally, and once (shown below), parameters are passed in esi and edi.

Is using esi and edi for parameters some known calling convention? Or Is that an optimization?

Here's the code I noticed:
[code].text:0041F060 ; vsnprintf_wrapper
.text:0041F060
.text:0041F060 arg_0          = dword ptr  8
.text:0041F060 arg_4          = byte ptr  0Ch
.text:0041F060
.text:0041F060                push    ebp
.text:0041F061                mov    ebp, esp
.text:0041F063                mov    ecx, [ebp+arg_0]
.text:0041F066                lea    eax, [ebp+arg_4]
.text:0041F069                push    eax            ; va_list
.text:0041F06A                push    ecx            ; char *
.text:0041F06B                push    esi            ; size_t
.text:0041F06C                push    edi            ; char *
.text:0041F06D                call    __vsnprintf
.text:0041F072                add    esp, 10h
.text:0041F075                mov    byte ptr [edi+esi-1], 0
.text:0041F07A                pop    ebp
.text:0041F07B                retn
.text:0041F07B vsnprintf_wrapper endp
[/code]

I was under the assumption that as long as their pushed on to the stack it doesn't matter where they came from.

I forget what it is exactly but it has something to do with constant value(s).

[quote author=Joe[x86] link=topic=16602.msg167771#msg167771 date=1176179194]
I was under the assumption that as long as their pushed on to the stack it doesn't matter where they came from.
[/quote]
I'm not talking about the function called there, I'm talking about where esi and edi came from. They aren't assigned anywhere in that function, they're assigned when calling it.

[quote author=Hell-Lord link=topic=16602.msg167774#msg167774 date=1176180396]
I forget what it is exactly but it has something to do with constant value(s).
[/quote]
Hmm, that's possible. But I've looked at this function before in older versions, and that never used to happen.

.text:0041F075                mov    byte ptr [edi+esi-1], 0

Apparently it is constant values then, because that appears to be setting [edi+esi] to (what will look like) a null string.

[quote author=Joe[x86] link=topic=16602.msg167835#msg167835 date=1176319343]
.text:0041F075                 mov     byte ptr [edi+esi-1], 0

Apparently it is constant values then, because that appears to be setting [edi+esi] to (what will look like) a null string.
[/quote]

No.  Look at the arguments to __vsnprintf and the documentation for the equivalent functions (pulled from MSDN documentation):
[code]
int vswprintf(
  wchar_t *buffer,
  size_t count,
  const wchar_t *format,
  va_list argptr
);
[/code]
edi is clearly storing the destination string buffer.
esi is the size of the destination buffer.
ecx is the constant input format string.
eax holds the varargs argument list.

The purpose of the line you quoted:
[code]
mov byte ptr [edi+esi-1], 0
[/code] should be obvious to you based on what those arguments are.
edi+esi-1 is the address of very last character in the output string.  This sets it to null, presumably to ensure that if there's a buffer overflow, the result doesn't overflow another buffer.

In any case, that doesn't really have anything to do with iago's question, which is - since when are edi and esi used in a calling convention?

@iago: I would guess that there was probably some kind of modified pseudo-inline fastcall calling convention, or perhaps it's a result of storing register values?

esi is the position is it not?

This is due to advanced compiler optimizations on module-internal functions, where the compiler knows of all callers of a particular function and can then apply custom calling conventions to allow for better register use across callers.

Functions that are `externally visible' will not have such optimizations applied to them.

Matt Pietrek authored an article providing a basic overview of these sort of optimizations as implemented in cl 13.  Some of the information in that article is dated and not entirely accurate now, but it conveys the basic idea.

In versions of cl prior to 13, there are a couple of instances where you might see strange calling conventions in compiler generated code.  For instance, for a number of releases, the `_alloca' special function was implemented using a custom calling convention using `eax' as an argument register and altering the stack pointer of the caller.  The various versions of `_SEH_prolog' also use custom calling conventions.

Aha, so it is a weird optimization. I'll definitely have a look at that paper.

Thanks!

Just for the record, is this similar to fastcall using different registers?

[quote author=Joe[x86] link=topic=16602.msg169191#msg169191 date=1179471490]
Just for the record, is this similar to fastcall using different registers?
[/quote]

no, not traditionally. fastcall uses ecx and edx to pass the first two arguments, and the rest are passed through the stack. sometimes you'll see fastcall use eax, too, but i think this is a borland optimization. in broodwar's case, youll only see fastcall using ecx, for the most part.

on x64 systems, though, most calling conventions, such as stdcall, fastcall, etc, are ignored by the compiler, and the convention it uses is similar to fastcall. it passes the first four arguments by way of registers. sounds like it'd make debugging much nicer.

edit: see this

[quote author=Joe[x86] link=topic=16602.msg169191#msg169191 date=1179471490]
Just for the record, is this similar to fastcall using different registers?
[/quote]
I'm not really sure what warz is talking about, although I think he either didn't read your post, or he's agreeing with you in an odd way. So I'll answer.

Yes, it's absolutely identical to __fastcall except that it uses different registers.

If you're talking about the __vsnprintf call, it isn't identical to fastcall at all.

Except that the defining characteristic of fastcall is that parameters are passed in registers, and that function passes parameters in registers.

He specifically asked if it's the same as fastcall but with different registers. Are you saying that's not true?

It is not a fixed calling convention.  The compiler takes a look at the function itself, and all callers of the function, and from there decides the best way to pass parameter values.  It is subject to changing completely at a recompile.

yeah, that's true.